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Introduction 

♦ 



TIlis vokmic rcprcscMils the first comprcluMisivc cfft)rt to 
sur\'cy MCiidcniic program approval niKl review in hii^hcr 
education. Although the evaluation processes themselves are 
hardly ne\v» perhaps neVer hefc^re have they held such far- 
reaching ctMisecjuences. hi increasing numhers» legishuors, 
administrators, and even faculty themselves arc calling ft)r ways 
of measuring educational effectiveness. 

Historically, student prt^tests of the talc sixties, combined 
with an inflating economy, helped pave an irreversible path to 
the country's more recent taxpayer revolts. Throughout the 
seventies, pleas for performance audits and other educational 
control measures could be heard in both state and federal 
legislatures. And on campus, where earlier attempts had tried 
to expand and decentralize administrative procedures, deans 
and budget officers inade an abrupt and dramatic about-face as 
they sought ways to balance their bc^oks and instill confidence 
in a system that clearly needed bolstering. Today, long after 
student-activist energies have dissipated, demands for account- 
ability in higher education cc^ntinue. 

Aljthough it is now clear that program approvjil and review 
arc hardly panaceas for all the ills facing colleges and universi- 
ties, they have most assuredly proved themselves as useful tools. 
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PiT>t'nU'cl Ikmx' is a pit turtM^f the iinpiu t aiul rcsulis of aciKk'inic 
pro^^rani waluanoii on rlic iiulivichials most alTettoJ iiaincly, 
stiuK'uls, (aiultv, aJininistratt)rs, irustccs^ aiul Mai^'-U'vcl sialT 
who arc clircLtly involvt\l in ovaliiation ol aiadLMnit/occupa' 
tit)tial proi^raius. 

Fhi-^ uiulcMlyiii^^ Iranunvork for this study relics on a survey 
tulviMi in h)7^^thiU involved a|)pro.\iinately otie-thivd of the 
eountry's posrseeouilary institutions. A total of 1,082 institu- 
tions were apprtniehech of which 68 percent respopded. In- 
cluded amon^ them were puhlic anti independent instiiucions 
with tW(vyear, four-year, anti graduate-level programs, hi addi- 
litui, \'isits were made to 53 institutions in 1^)77 and 1980, based 
on their types tif evaluation programs (including both ^'modeP* 
and "typical" cases) as well as their availability hv scheduling. 
Some site visits ciinsisteU of little more than evaluation' of a 
single aspect ov procedure, whereas others consisted of in-depth 
interviews with students, faculty, administrators, state-agency 
staff, board members, legislators, and others. 

As might be expected, many of those interviewed tended to 
view program evaluation from the perspective of how it affected 
thcni personally and pr()fessionally. For some, program evalua- 
tion was seen as a process designed exclusively for program 
improvement; for others, it provided a means for setting institu- 
tional priorities and resource allocations; and for still others, it 
was simply another euphemism for program curtailment or 
elimination. Yet time and again, our findings revealed that the 
picture is far from being either all black or all white. Not 
everyone fits into the polarized camps. In fact, the majority 
probably hold mixed views, feelings, and reactions. 

Regardless of the views we may hold, present demands for 
accountability will doubtlessly lend an added importance to pro- 
gram evaluation throughout the coming decade. Obviously some 
kind of mechanism is becoming increasingly necessary for making 
sure that budgets get balanced, educational outputs remain pro- 
ductive, and rcvsource allocation decisions are rationally madeJ 
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IVvhikI cost savii)^ considcraiions ;K;KU'inic proi^rain 
evaluation Wvs a Jn'por coiucm ii for iMMiriiij^ tho inluaviit i|iialiiv 
of (Hit' posiscv oiulary institutions. Suruly^ otio of tlu* tnori' ijjrati- 
lyin^ aspi\is ol the pivscnt study is to witness ilu" iiureasiiHi 
sophisticaii(,>n ol evaluation proceihires. 11' our priuun).^ can hv 
clone in t itn(s selev t in^ the strei^^t hs aiul weaknesses olai a Jem ie 
programs, perhaps we ean avoiJ the disastrous results of inakiiHi 
aeross-the-hoarJ etus later on, Ailiuinistrators wlu» ituist 
allocate aiul real loi ate in t itnes of ecotiotnic scarcity ean n^u hope 
to make their Jecisions wisely utiless those consiJerations in- 
cluvle t horouv^h aiul systematic jM'o^rain reviews. AtuI similarly, 
those pro^^rams that pass throui^h the most rif^orous approval 
[M'ocesses will surely he the most apt to survive the itievitahle 
trials ol eiuollmetit declines ami fiscal austerities further down 
the roatl. Ouv present fituliuj^s repeatedly indicate that although 
weak proi^rams tnay suffer, strong proj;rams are simply made 
stronger hy iiiKler^oini^ evaluation proeedures. 

Thr(ui[^lunit our investigation, material has heen ji^athered 
that is felt to hec^f pariieular S'alue to practit ioners in invstsecond 
ary education, (lenerally speaking, examples were selected 
because they typifietl the kinds of policies and procedures utilised 
in a certain context; they are nor intended to serve either as en- 
dorsements i^r models for others to direetly emulate. Individuals 
wanting to develop or revise an approval or re\*iew process are 
advised to piek and ehcn^se elements that best meet their unitjue 
reciuirements. Perhaps it should ^o without sayinjj; that although 
many proeesscs wc^rk effectively within the context where they 
are being implemented, none is applicable to all situatiotij^. 

To relate our findings in the most succinct matiner, every ef- 
fort has been made to limit technical terminology to a very few 
critical concepts; to prcu'ide charts, tables, and graphs otdy 
where necessary; and to explain by way of example whenever 
possible. 

At this point, the reader who is not thoroughly familiar with 
the language of prt^grain-evaluation procedures is strongly 
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encouraged to refer to Appendix I, entitled "Terms and Defini- 
tions.** Without a complete understanding of the crucial distinc- 
tions separating various kinds of review and approval activity, 
erroneous conclusions can easily he drawn. Also, for those seek- 
ing further information on aspects of program evaluation, the 
selected hihliography at the conclusion of this study is included 
as a helpful resource. 
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CHAPTER 1 



Approving 
Programs Internally 



In the late 1940s, Professor L. Bailey (an assumed narne)^ 
decided to unite his vocation with his avocation. As a pro- 
fessor of horticulture at a fairly large land-grant university in 
the Midwest /Professor Bailey was in an ideal position to ex- 
pand his fascination with the art of ornamental horticulture 
into an undergraduate degree program. With just two students,, 
a portion of an existing greenhouse, some donated equipment, 
and an unused plot of farmland owned by the university. Pro-- 
fessor Bailey had, within a montlVs time, established a program 
in ornamental horticulture. Major curriculum for the program 
consisted of courses, texts, and supervised learning experiences, 
all of which were developed and taught by Professor Bailey with 
the part-time assistance of a colleague from the Agriculture 
Department. Degree arrangements were easily finalized by the 
registrar's office. 

Since the 1940s, times have changed drastically. If Professor 
Bailey were to establish his program in ornamental horticulture 
today at the same university, he would need first to obtain 
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departmental approval for his proposed program; the proposal 
would then go , to a curriculum committee of the College of 
Agriculture; and then, if approved, it would be forwarded to 
the universitywide curriculum committee. After this, the pro- 
posal would be forwarded to the appropriate university admin- 
istrator, and then, if endorsed by the administration, it would 
be sent to the board of trustees for their acceptance. But the 
process would not. end there. 

In all likelihood, the program-approval process would con- 
tinue from the university to the state postsecondary planning 
committee, which would review the program and make a 
rect^mmendation to the planning commission. Finally, the 
ebmmission would either approve or deny the request. 

Depending or\ the availability of financial resources. Professor 
BaileyV proposal might also need to go to the legislature for ' 
funding, where the proposal would undoubtedly require further 
review. In developing his program proposal today. Professor 
Bailey would need at least two fairly comprehensive sets of 
documents (one for internal use and one for external use), pro- 
viding extensive data, on projected enrollments, equipment 
needs, staffing needs, and funding. Industry and manpower ] 
' projections would be required both to confirm program needs 
.and to demonstrate clearly that his proposed program did not 
unnecessarily duplicate efforts at other state institutions. Fur- 
thermore, he might also be required to have outside consul- . 
tants (usually out-of-state peer consultants) review his program 
at one or more levels. Unlike the month that it took Professor 
Bailey to establish his program in the 1940s, it would take about . 
a year and a half at a minimum for his proposal ro obtain final . 
approval today. 

Professor Bailey's hypothetical experience can be viewed as 
symbolizing the complexities of modern-day higher education. ■ 
Also, it demonstrates a typical series of procedures necessary for 
obtaining program approval. Unlike the "golden era of expan- 
sion," which influenced higher education as recently as the 
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1960s, increasingly demanding efforts are now required to. ob- 
tain increit^M^ly dwindling resources. At nearly every public 
college or university in the country, a series of formal internal 
processes an\i procedures is required for obtaining approval of 
proposed programs. In addition, ai least one external group of 
procedures is called for at the final approval stage. In all but 
->hout five states, a postsecondary-education commission 
reviews and approves programs. Although state agency reviews 
.typically affect only the , blic institutions, independent col- 
leges and' universities that depend on public aid increiisingly 
find themselves subject to similar regulations. Without detailing 
.the conditions leadi'ng up to present circumstances, suffice it to 
•say that program approval is the child of a labyrinthian educa- 
tional system, a system that expanded speedily in the latter part 
of this century. 

Focus in this chapter centers on program' approval within col- 
leges cir universities; in the next chapter, focus is on system- and 
state-level review process. Indeed, because of the abundance of 
state-level postsecondary agencies, program approval is typically 
a continuous process running from the proposal development 
stage at the department or faculty level through final program 
approval at the state level (see figure 1). 

Internal program-approval procedures are found today in 
•irtually all public institutions, as well as in most large indepen- 
dent colleges and universities. According to our survey, formal 
procedures. are increasingly utilized (in their order of frequency) 
in independent four-year liberal-arts colleges, private junior col- 
leges, and proprietary schools. Only a few institutions— mostly 
small private colleges — have no procedures whatsoever.. 



The Reasons 



Almost bll internal program-approval processes seek in one 
way or another to improve institutional administration. Dif- 
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FIGURE I 

Typical Program Approval Process for Four-year Colleges and Universities 



IX'pt, Ap|)ri>\'cs 

I-aciilty 
Proposal 



Colli'giatc- 


Yes 


College or 
University-wide 
Review 


Yes 


Faeulty 


Yes 
— 


Cnmnutict* 


No 


No 


Governance 
liody Review 


No 




Committee 







Academic Affairs; 
Dean or V.P. 
Review 



(Notice o( Intent to Develop Proi^ram) 



No 
-4— 



Admin. <S<. Board 


of Trustee 


Approv 


^\ 


Ve5 




No 




r 






State 




Post secondary 


Agency 


Staff Review* 








Yes 




No 




f 






State 




Boarc 




Review 



o 
cr 



7r 



*lnsert system review here if institution proposing the program is a part of a system'of institutions (that is, a mulricampus institution). 



17 



T!':r r f :■■ 



APPROVING PROGRAMS INTERNALLY 



fcTLMiccs exist only in detail. Basically, they try to provide a 
systeniatic means for controlling expenditures while simultane- 
ously meeting the needs of Cjuality assessment. Unlike business 
or industry, however, higher education typically operates 
withiHit a strt:)ngly centralized management structure. Although 
this prcKess no clouht contributes to a more democratic form of 
dccisi(Mimaking, it is wry time consuming. With few exceptions, 
most colleges and universities utilize" some form of cjuasi- 
centralized tir collegial administration in matters relating to pro- 
gram approval. And interestingly, the exceptions are usually 
found only in two-year colleges, bible colleges, and proprietary 
•ihstirutions, which are usually closer in management style to 
[private business than tc^ typical colleges and universities. 

ReasiUis cited for conducting internal program approval are 
fairly similar in most institutions under the Carnegie classifica- 
tion system (see figures 2 and 3). Heading the list is a desire "to 
deterTnine if clocumentecl needs justify the program," such as stu- 
dent ini:erest, employment opportunities, and so on. In order of 
frequency, the second and third most popular reasons are: a 
need "to determine if resources (are) sufficient to support a qual- 
ity program," and a desire "to determine if the program is consis- 
tent with. institutional role and mission." 

In additicni to the above 'Uated reasons, yet another reason for 
formal internal program approval surfaced frequently in in- 
depth interviews. Perhaps this purpose can best be stated in the 
■words of a university program-approval report, which forth- 
rightly explains that programs that have been carefully scru- 
tinized on the campus act "as a control against inclination at the 
state level to select prc^grams for funding in amorc political way." 

Public colleges and universities in particular indicate a belief 
that by doing their internal program-approval' homework they 
may best be able to forestall politically motivated external 
approval activity. In this sense, "political" refers to unwarranted 
intrusion into the areas of institutional autonomy and academic 
freedom, as. well as to the distribution of resources and programs 
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FIGURE 2 



Major Purposes in Conducting Internal Reviews of 
Proposed New Programs (Percentage) By Type of Institution 
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FIGURE 3 

Criteria for Approving New Program Proposals 

(% of Total Itistitutions and 
Rank Order of Responses to Each Criteria) 
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to the various state institutions. Although the evidence fails to 
indicate conclusively whether or not internal approval processes 
actuiilly curb or limit external reviews, there can be little doubt 
that most external reviewers appreciate the fact that internal 
approval procedures can help lighten their loads. 

Aside from positive aspects, how(^v(T, r(^mpr(^hensive internal 
reviews can have negative implications as well. In one instance, 
the state agency fail' d to differentiate between institutions with 
thorough internal review procedures and those with poor or 
even no internal approval procedures. In this case, the latter 
category of institution had an obvious advantage since a greater 
percentage of their programs were passed along to the state 
agency. In a related situation, the state agency kept score of how 
many programs it approved and disapproved for each institu- 
tion. Under this system, no matter how good a job an institution 
did in internal review, some of its proposed programs still were 
turned down, simply to make it appear that the institution was 
receiving fair treatment. A problem emerged with another such 
scoring practices in a state where all the weaker program pro- 
posals were weeded out before proposals were sent to the state 
higher-education agency. In this state, because only a few pro- 
grams were ever turned down, a key legislative committee not 
only determined that the state agency wasn't doing its job but 
even went so far as to try to establish a legislative program- 
approval process. Obviously, in all these instances, qualitative 
dimensions were obscured by a preoccupation with program 
rejection rates. 



The Participants 



Participants in the program-review process tend to hold widely 
differing views. For example, program faculty may be cither 
critical or positive about a particular proposal, depending on 
their ability to influence the results. As fir.ancial resources have 
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dwindled, it's become more and more likely thalE if one group of 
faculty gets a program, another group will have to do without. In 
some locations, the state postsecondary-education commission 
actually requires tradeoffs, where one program must be dropped 
for every new program added. Not surprisingly, some campuses 
visited during the course of this study reported vicious conflicts 
cn-er program approval and resource distribution, a situation 
that is particularly apt to occur on campuses that are heavily 
politicized by rival faculty unions or factions. 

On another level, nonprogram faculty who sit on review com- 
mittees have an awesome responsibility to maintain impartial 
views on program proposals. While most faculty committees we 
met with undertook thorough reviews, some exhibited a ten- 
dency to pass on difficult recommendations to the campus 
administration, only to turn around later and criticize the 
administration's decisions. As program faculty become more 
adept at proposal development, it becomes that much more dif- 
ficult to select among competing proposals. In response to such 
perplexing alternatives, one faculty committee even went so far 
as to reject an attempt to introduce consistent criteria to their 
review process. Instead, they opted to send all proposals on to 
the administration to "let them make the choices." Although the 
extent of the problem is difficult to assess, we found several 
instances in the .course of this study. 

In many cases, the task of approving or disapproving new 
programs ultimately falls to the dean, the provost, or the vice- 
president for academic affairs. According to our survey, this per- 
son's role is incre^asing on many campuses for the above-noted 
reasons, as well as due to a more general trend toward greater 
administrative control. But if this trend is relatively new at larger 
public institutions, it*s long been the policy at many community 
colleges, bible colleges, and proprietary institutions. At these 
institutions, the dean has always been the person primarily""" 
responsible for program appro^l, and frequently he or she is 
even^the developer of the initial proposal. As a case in point, we 
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'found one bible college in the course of this study that did not 
even have a faculty curriculum committee. All curriculum deci- 
sions were made by the administration. 

Besides faculty and chief academic officers, a number of others, 
such as trustees and students, take part in the program-approval 
process (figure 4). Generally speaking, as far as trustees are con- 
cerned, the larger the institution the smaller their role. Yet 
trustee attitudes, combined with those of the chief executive offi- 
cers, often determine how vigorously program proposals are in- 
ternally reviewed. Student roles, on the other hand, tend to vary 
widely in all types of inst'tutions. Although many institutions 
try to include students in the approval process, actual student 
impact depends greatly on the opportunity of students to partici- 
pate in the process. Among those institutions with student in- 
volvement, three primary student activities are identifiable: 
membership on an institutionwide committee (with or without 
vote), involvement in program development at the departmental 
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level, and participation in surveys. Of the three, membership on 
institutionwide committees held the most influence and, inter- 
estingly, both fnculty and administrators at institutions using 
this approach seemed generally positive about the result^. 



7 he Procedures 



On some campuses, program-approval procedures are formally 
written out and widely distributed. On others, the procedures 
are less formab v/ith the specifics for each proposed program 
determined individually. Based on our findings, the larger the in- 
stitution, the more formal the procedures. But particularly at 
large research universities, these formal procedures tend to main- 
tain a high degree of flexibility. For example, the University of 
Minnesota procedures state that "the University does not 
prescribe the form in which the proposal document must be 
cast." Instead, they offer a basic outline "organized on the con- 
tours of past proposals which have been communicated effi- 
ciently and with fidelity: . . By way of contrast, other types of 
institutions with such formalized procedures tend to be much 
more prescriptive. In fact, as a time-saving measure, they 
sometimes merely incorporate or supplement the procedures of 
state coordinating boards. Nonetheless, this method may work if 
strictly institutional concerns are not slighted by state officials. 

At this point, to help the reader better understand some of 
these methods, we might look briefly at a specific program- 
approval process such as those indicated in the flowchart from 
the University of Alabama (figure 5). At the Birmingham 
campus, the graduate program-approval process involves deans, 
the Graduate School of Administration, various committees, 
the system-level office, the board of trustees, and the Alabarpa 
TTommiSion on Higher Education (ACHE). 
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FIGURE 5 



University of Alabama Graduate School 
Submission of a New Program Proposal 
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Community Colleges 



Up to this point, Jiscus^^'on about internal program approval 
has centered primarily on four-year public colleges and universi- 
ties and large pri\'ate institutions. Community colleges exhibit a 
number of interesting variations worth investigating. Perhaps 
foremost among these distinctions is tKeir relatively heavy 
reliance on manpower studies for program initiation. Unlike 
four-year colleges and universities, which often propose pro- 
grams with vaguely defined potential job markets, most com- 
munity colleges emphasize strong employment prospects. In fact, 
sometimes graduates are virtually guaranteed specific jobs.* 

To illustrate the extent of such efforts, one might note the pro- 
cess at Gateway Technical Institute in Wisconsin. At GT,I., a 
proposed program must meet the qualifications of something 
called "Manpower Demands," which examines present and 
future job possibilities. One aspect of "Manpower Demands" 
focuses on specific "job outlook" levels necessary for degree com- 
pletion, including detailed job descriptions; diploma-level jobs; 
job-placement locations (such as the size, scope, and location of 
labor-market demands); and eventual promotion levels. 

Unless requested to do so by an external agency, few four-year 
institutions document manpower needs in this* detail. But re- 
quirements are changing. Many four-year institutions are start- 
ing to ask for and get better job-prospect documentation. In 
some cases, this trend is a natural response to criticism about 
graduates not finding jobs. Yet, because manpower projections 
seem to be. more reliable in the vocational areas than in-other 
kinds of educational programs, our respondents in four-year in- 
stitutions often expressed skepticism about-utilizing marketing- 
type procedures for program development and approval. Figure 
6 indicates a hypothetical approach that makes use of a weighted 
system for determining program marketability. 
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Another distinguishing feature in community-college proce- 
dures is their tendency to defer program decisions to administra- 
tors rather than faculty, aifinding often noted by others studying 
community-college adniinistrative styles (McCord 1974; 
Richardson and others 1^72). To illustrate the point, let us look 
briefly at Broome Community College in New York. At 
Broome, program proposals originate in campus-offices, such as 
those of academic personnel, continuing education, placement, 
office, or the curriculum dean. They are then reviewed by the 
curriculum dean and sent on to the appropriate line administra- 
tor. Copies of the proposal are also forwarded to the director of 
planning and development, as well as to the college senate and 
the Curriculum Committee. Finally — after a successful feasibility 
study— the curriculum dean and the vice-president for academic 
affairs send the proposal on to the president. Assuming approval 
at each stage, .the President 's Council will review the proposal 
and make a recommendation. If the president gives the go- 
ahead, the proposal is sent first to the trustees, then to State 
University of New York Central, and then on to the New York 
State' Education Department. 

Although specifics may vary from college to college, Broome's 
program-approval process is fairly typical of the strong adminis- 
trative influence at most two-year institutions. Moreover, the 
degree of administrative involvement in the approval process is 
consistent with the generally heavy role of community-college 
administrators in virtually all other aspects of community- 
college operations (Thornton 1^66; Blocker and others 1965). 

Another typical aspect of Broome's approval process is its 
reliance on a support unit, in this case the director of planning 
and development. Community colleges, much more than their 
four-year counterparts, utilize staff units. Although at Broome 
the director of planning and 'development merely coordinates 
the feasibility assessment, in other community colleges support 
units may actually perform the feasibility studies themselves. 
Services of these support units range from simple coordination 
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and basic ^research to taking full responsibility for proposal 
development and even playing a key role in final approval. 
Similar units may be found in most four-year institutions, but 
their actual program-approval role is generally more limited. 
. In four-year colleges or universities most (if not all) program 
proposals are faculty-initiated, but at two-year institutions faculty 
are just one of many such sources. For example, the Community 
College of Philadelphia notes the following new program 
sources: 

1. Personal observations on the characteristic's of the local 
and regional labor markets 

2. Manpower-studies data 

3. Requests from industry 

4. Requests and suggestions from students, faculty, adminis- 
tration, and board 

5. Recommendations of college advisory committees 

6. Offerings at other schools " 

In another instance, a policy at Williamsport Area Community 
College in Pennsylvania states that the college seeks "suggestions 
for new curriculum(s) from its broad constituency of students, 
faculty, staff, sponsors, trustees, business and industry, trade and 
professional groups, government, and the larger community of 
citizens." 

Because of the nature of these sources, administrators arc apt 
to play a more central role in program initiation than faculty. 
Consequently, the internal approval process in the community 
colleges tends to be considerably shorter than at four-year insti- 
tutions, simply because administrators have a hand in both ends 
of the process— initiation as well as approval. 

Another interesting difference from four-year institutions is 
the role local citizens often play at community colleges in initiat- 
ing, approving, and even terminating . programs. Citizens 
increasingly help gear program designs more directly to ^com- 
munity and business needs by serving on advisory councils and 
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FIGURE 6 

Summary of The Academic Program Development Procedure 



1 

• nffinition of 






irdOcncrutlttn 


Minion iUiJ 




N 


ilu- ulr.i 


Scrvici' Area 




Ji 


>nli liinlu-r 
\flii|tim'Mt.' 



I 

Ufil Scri'i'ninK 

Is tUv ultM 

in«tlttiM|ili;)l 

nicMtm .ukI 



» C\tncf|il 
l/fvclopnu'nt 

C'.lt) till' ultM 

In- Oi-u-h-iH-J 
ititt< ill) ;ip|v.ilm« 
priti;riini tniKt.'|M,' 



I" till- prnnfiiiii 
.itul :ip{H'.iliii^* 

.tViMil vlllplk .UKin 

of ..thrr-' 

I- lUIt-lilf 

liliKhiK ,iv;iil;tMf.' 







■ SlicukI ulf.i Ih- 

".flit liiU k f*u 

ilt'\i'Kiptiu'nt? 







Prt)|{r<iin Dvitiami 
l>< ilu-rc siillu ii-ni 
Jt'ni.iiul Itii till' 

plO^l.tTII.' 



• Prll^ram 
|\va)uation 

SliMuM tlir 

i[npl(>tii«'iiii'«l.' 



I'xp-rt 1 iinitntiniiy 
IciJiT (i|iiniiin 



M.wklt Ml 


\i'V 


ri-vi'.Uk li 





1— Miirkit IV.t 



Drup I Jca 



(StlfKCH; \X*iiviK' A. I.t'f iuul JnH'ph li. Oilnunir, )r. "A rfiKfJun- for iIk- IVvvK^pnu-iu nfNcw PriYrjjmsm ro^Koiomliuv {;iliuaniin.".f«>»nuil)>/ HikHi'i KiitudHon. Vol. M.VIll, No K M.»v ')i»nc 1**77 



. W7 ) 



29 



BEST COPY mm. 



APPROVING PROGRAMS INTERNALLY 



committees. An appropriate example can be found at Housa- 
tonic Community College in Bridgeport, Connecticut, where 
advisory-committee members are selected in accordance with 
standards similar to those found ]n fair-employment practices. 
Once appointed, these citizen participants not only assist in pro- 
gram planning and evaluation but also help maintain regular 
contact with the occupational community. Often, they are 
called in when program needs first arise, and more and more 
they stay on to help actively with program needs. Although 
many four-year colleges and universities now make use of advi- 
sory groups, they usually play considerably reduced roles. 

In addition to the above-mentioned aspects, two-year colleges 
distinguish themselves from four-year institutions by means of 
the very quantity of new programs they propose, especially in the 
occupational area (see Cosand 1979). Partly, this reflects the 
short-term, manpower orientation of community colleges and 
partly it reflects the dramatic recent growth of community col- 
leges. By 1978-1979, 36.4 percent of postsecondary students were 
enrolled in two-year institutions, according to the National 
Center for Education Statistics. 

To return briefly to the case of Professor Bailey, it*s difficult to 
say. whether or not his ornamental horticulture program would 
find a niche today in one of the country's community colleges. 
But at least in terms of the red tape involved, he would probably 
receive his answer much faster there than at most four-year coU 
leges and universities. 
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Approving 
Programs at 
System and State 
Levels 



System-Level Program Approval 



Program approvals typically continue well past the level of 
internal approval. The next step for an increasing number 
of institutions, and particularly for community colleges, is the 
system-level review. To date, however, a common complaint of 
system-level review has been its failure to see beyond its own 
immediate horizons. For example, in states where postsecond- 
ary education is divided into subgroups (such as community 
colleges, four-year colleges, and research universities), the 
system perspective is often no wider than the segment it 
represents. As a result, program duplication occurs easily. 

All of the system offices we visited conducted some kind of 
program approval, and many system (and district-level) offices 
are actively upgrading their effectiveness. In most cases, system- 
level reviews are conducted by interinstitutional committees 
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comprised o( nciKlcmic officers from constituent institutions 
and sy: tenvlevel staff. Less frecjuently, reviews are entirely con- 
ducted by staff. But whatever the case, these groups usually 
niiike recommendations to the district or system hoard for 
either approval or nonapproval. 

As a specific example, we might look at the Minnesota state 
university system, which illustrates the role of a multicampus 
hoard in approving undergraduate programs. By utilizing an 
interinstitutional committee and external consultants^ the Min- 
nesota state university system provides a fairly typical example 
of a university systenvlevel office. According to the Minnesota 
procedures, university-approved program proposals must be 
sent to the vice-chancellor for academic affairs, who in turn 
send.^ copies to pertinent agencies, such as the Department of 
Education or the Higher Education Coordinating Board. The 
vice-chancellor's office then prepares a proposal summary for 
the state university board's Educational Policies Committee. 

At this time, the university president and others may appear 
before the committee to react to the summary or to elaborate 
further upon the proposal. Even if a program proposal has been 
denied by one of the consulting agencies, the Educational 
Policies Committee retains authority to override the decision 
and permit trhe university to implement the program. 

Although it's the responsibility of the vice-chancellor for 
academic affairs to keep the chief academic officer of the univer- 
sity informed about the status of the proposed program at each 
level of consideration, it's the job of the chancellor to actually 
approve certain new programs when the impacts of such 
changes fall within certain prescribed guidelines. If the 
chancellor has questions about any of the proposals that fall 
within his or her jurisdiction, or feels that a program should be 
further discussed because of its implications for educational 
pblicy, then the proposal can be deferred to the committee. In 
like manner, if the university president disagrees with the 
chancellor's decision, the matter can.be arbitrated by the 
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cointnittcc. With slight variations, cntailinj^ the use of cnitsido 
coiisultatns, a similar prtHcdurc is devised (or ^raduatclcs'el 
program prtiptisais in Minnesota. At tliis point, heeause details 
o( systenvlevel reviews are reniarkahjy siniihir to siate-level 
operations (Smith 1^)80), our attention will now foeus on the 
speeifie workings of state-level approval. 



State- Level Program Approval 

Generally speaking, state-level program-approval responsibilities 
have evolved hand-in-hand with the agencies themselves. Many 
agencies began as voluntary coordinating boards and gradually 
acc|uired broad discretionary powers cither as regulatory boards 
or, in some instances, even as statewide coordinating boards. 
Today, program-approval responsibilities of these agencies are 
still growing, as evidenced by recent statutory changes in several 
states (figure 7). 

By 1981, stale agencies or university systems in all but 7 states 
conducted at least some new program approval for public institu- 
tions. And out of the statrs represented, 9 also conducted new 
program approval for private/independent institutions. In terms 
of their actual responsibilities, 33 had authority to appro'/e pro- 
grams in the public sector, and 10 could review and make recom- 
mendations. Interestingly, only 5 agencies had authority to 
actually approve in the independent sector, but 4 could make 
recommendations and conduct reviews (figure 8). 

By contrast, a, 1978 study by Barak and Berdahl found that 
agencies. at that time were still very much in the developmental 
stage. In that study, 21 states reviewed new program proposals; 
today, 47 states exercise that responsibility. Similarly, a number 
of agencies that previously only had authority to make recom- 
mendations now have increased regulatory responsibility. Many 
of the agencies that formerly had relatively weak statutory 
responsibility now have added clout by virtue of their expanded 
budgetary '.esponsibi-iities. Clearly, the final determination for 



27 



33 



00 



riGUUli 7 

Status of State-Level HiRher Education Agency ProRram Reviews 

(New Pro|»r{ims) 




I I Sinio t'lMului'iinn fn^riiin Jipprttvitls 

StJito Jt , ' iuK [>roj:r.)iii npprovul prtKcss 

Pro^jrnrn np[)rnviil ci'^'.'luctt-J by respective uiiivcrsitv/ftillcjic system 



O 



(SOURCE: Robert J. Bnrnk^ .Stt/tc-Lael HiRhcr' Education Pro«non R.viVtc UpJdfc: 1979, SHEEO/NCES Ctimmuiiientitins Network, 1979.) 



3,^ . BEST COPY Ai/AiLABLE 



ERIC 



APPROVING ImToG> AT SYSTEM & STATE LEVELS 

nuw prt)^rnni pniposals hns shifted uut of the hniuls of the iiv 
stiiutioiis iuul into the stiue-level postseeoiulnry ajj;eneies. 

Ill the Barak iiiul Rerclahl study, ccMisiderable eoncern was 
cxpressecl about which new proj^raius should be reviewed at the 
state level ancl how they shcnild be defined, but this no longer 
appears to be an Issue. A program now is largely viewed as *'a set 
of courses or offerings leading to a degree or certificate," with the 
exact definition \'arying from state to state. 

FIGURE 8 
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Authority to Approve at Least Some New Programs 
By Governance Type and Sector 
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As might be expected, the scope of programs under review has 
clearly broadened. While some states formerly reviewed only cer- 
tain specified programs, such as upper-level graduate programs, 
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niiisi siaic M^LMH'U's tunv review nil new pro^rjuws, and many 
appnnx' pri)jL»rain majiM's as well as tiuniu's. In a similar vuin, advli- 
liimal siM'vii e and luureaeh pn^^i'^^ii^^ l^'^^'^' joined the lisi o{ pro- 
^raiiis that eome under review, 

An analysis o( rhis e\'olulion seems to show an inerease in 
state-level ageney involvement following a proportionate 
deerease in institutional eredibility in the eyes of many state 
legislators and executives (Geio^ue 1^\S0). Although credibility 
los^ in some instances, is unrelated to academic programmatic 
issues— as through an athletic scandal or student unrest— added 
restrictions are often seen as a way government officials can in- 
crease institutional accountability (see Gentile 1980). Admittedly, 
state officials are sometimes plainly frustrated by the seemingly 
never-ending demands for more funds. Whatever the case, state 
agency pcnvers are increasingly used as a means for keeping better 
tabs on resource allocation. 

Although actual review criteria such as needs, costs, and tiuality 
control have not changed significantly at the state level, agency 
sophistication in applying these criteria has. No longer are single 
measures or indicators used in applying review criteria. Today, 
agencies use multiple criteria, ask more difficult questions, and 
demand more exact responses. Several states now ask for 
program-performance measures to check up on programs after 
the initial approval. , . 

Partly in response to criticism, the state-level approval process 
has recently become more open to institutional participation in 
decisionmaking. Also, some state agencies are now defining their 
territory more clearly to encourage certain kinds of institutional 
program development while implicitly or explicitly discouraging 
•others. For example, the Department of Higher Education in New 
Jersey encourages the development of new academic programs in 
industrial, technological, governmental, and human-service 
manpower concerns. Also, they encourage community-college 
initiatives "in such high technology programs as machine tool 
technology, aviation maintenance technology, and energy 
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;icchni)K)jj;v . . . OiluT pri)|j;rams» iif course, arc inipliciily 

Several siaiy a^eiu ies m)\v require (either loriiially or inform- 
ally) some foriii ol a **starl one-slop one" appr^^aeh to new pri^- 
[^rain approval. In ihose states, insiiiuiions are expeeteJ ro 
lernunare a pro^zrani each time a new program is proposed. 



Some Typical Statc-Ajj^ency Procedures . 



Two basic approaches arc employed for statC'level program ap- 
proval: (I) planning and/or budgeting, and (2) incremental. 
States using the planning approach usually request explicitly 
that new program proposals be coordinated with an overall plan- 
ning effort, generally as a part of the state s master-planning pro- 
cess. In New York, for example, new program proposals rhust 
first meet the reo'iircments of the state's master plan and only 
later are they referred for specific program approval. Incremental 
approval, or the one-at-a-time method, is no doubt more com- 
mon today. In these cases (except in states where all proposals 
must be turned in at a particular time) program proposals arc 
submitted at the convenience of the proposing institution. In 
other words, the relationship to the planning process is con- 
siderably less explicit in states using this approach. 

Advantages and disadvantages are found in both approaches. 
Where some institutions have criticized the planning approach 
as being too lengthy and inflexible, many state agencies teel 
delays are justified when proposals can then be coordinated with 
master-planning and budget needs. In fact, even states using the 
incremental approach often do so within the general context of a 
master plan, while retaining program approval and planning as 
separate processes. Moreover, many states using the planning 
approach provide additional flexibility by allowing institutions 
to modify the planning process in special circumstances. To get a 
better idea of the actual implementation of these approaches. 
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profiles of ScIlvcclI prognmvnpproval processes arc iiu'ludcd in 
Apprndix II. 

hi co!ulusion» it should he nofed (hat u\^o-ycMr community 
colleges and vocaiionahicrhniral schools often have parallel 
and, unforiunaiely» sometimes rcLlundanl state-level approval 
procedures. Some slates offer both a state-level postsecondary 
coordinating/governing commission, as well as a separate state* 
level hoard for comnuiuity-cc^'ege and vocational-technical 
education. Understandably^ when this dual structure occurs, 
many agencies try lo combine their respective approvals into a 
single effort. 
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CHAPTER 3 



Reviewing 
Programs Internally 



Current Trends 



Perhaps one way program review ean be distinguished from 
program approval is by the arhount of controversy that 
surrounds it. When resources must be reallocated, courses 
canceled, or programs and faculty terminated ,'-the review pro- 
cess often becomes a topic of fervid debate. Newly proposed 
programs^ on the other hand, frequently do not have faculty, 
students, or resources already committed, so their nonapprovnl 
tends to be markedly less controversial. 

Des|..te the recent disputes, the topic of internal program 
review is hardly new. Many postsecondary institutions— in one 
form or another— have been reviewing their programs for years. 
While early surveys of program-review activities are notably 
scarce (Mortimer and Tierney 1979), this doesn't necessarily 
mean that reviews didn't exist. Daniel Stufflebean (1980) refers 
to an unpublished survey that shows limited review activities at 
several universities in^the early sixties. Our survey results 
substantiate this finding. Approximately 12 percent of the 
responding institutions indicated their program-review policies 
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nnJ pnvcJurcs hc^an pric^r \o \%5 (sec fi^uri' ^)). Whal is new, 
lunwvrr, is tlu* Jrainatii* ituTcasc iu prc^jjjranMTvic'W activiiy 
sin(c llie* initi lo early si'vciuics, 

In h^7^, du.' C^arnr^ic C'ouiu'il speculated ihai insiiiutional 
aelininisiratt)rs wcnikl he relying u\w.h more heavily on pro- 
gram and eiHirse review in the future (Glentiy and others 1976). 
Their predictioi\s were overwhehninjj;ly aecurate. Approximately' 
76 percent of the institutitins we surveyed initiated their present 
policies alter 1^>70, and morethai^ halfof these (4 ^UM'cent) said 
their reviews were developed after 1975. Today, the trend is still 
growinj4, More and n^iore colleges and universities are unclertak' 
injj; internal program reviews or are planning to do so in the 
near future (Glenny and others 1976), 



Date Present Policies and Procedures Initiated (%) 
In Cotteges and Universities 



Before 1965 1965-69 1970-75 Since 1975 

12% 12% 33% 43% 



Out of all of the colleges aiid universities surveyed, 882 (82 per- 
cent) said that they employed some form of **formal review/* 
which, in limited cases, might be taken to mean an annual budget 
review or even a regional accreditation review. Interestingly, 
many of these institutions reported that thcy had internalized 
their state*s review procedures as their own. A few institutions, 
22 (2 percent), responded that while they did not have a formal 
system of internal program review, they did have an informal 
process. And finally, about 178 (16 percent) of the responding 
institutions indicated that they conducted no internal program 
reviews whatsoever (see figure 10). 
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FIGURE 10 



Colleges and Universities, by Type of 
Program Review Activity 



, Number Percent 
Have Formal Program-Review Process 882 82* 
Have Informal Program-Review Process 22 2 
Did Not Conduct Internal Program 
Review 178 .16 

TOTALS . 1082 100% 

(R. j. Barak 1980) 

NOTE: Because the survey respondents were not all responding to the same 
definition of "program review," varied institutional processes (such as an 
annual budget review and a regional accreditation review) were solinetimes in- 
cluded. Accordingly, the actual number of institutions with '^stematic 
evaluations of all academic programs is presumably less than the 82 percent 
indicated. 

Our survey responses clearly indicated that large research 
universities^are most apt to conduct internal reviews. And 
perhaps coincidentally , universities that rank high in the various 
peer ratings are likely to ohow a similarly high usage of internal 
program reviews. In some cases, however, the reviews are strictly 
limited to graduate programs. 

Unless the four-year type of public institutions or community 
colleges are members of systems or districts that either encourage 
or require reviews, they are less inclined to conduct such evalua- 
tions on their own. Nonetheless, a noteworthy exception to this 
survey result was found in Community colleges that conduct 
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program evaluations in order to receive grants under the 1976 
Vocational Education Act. Other types of institutions in our 
survey, such as predominantly black colleges and universities, 
bible colleges, small independent colleges, and proprietary 
schools, basically adhere to a pattern of either limited or no 
internal program-review activity. 

In those institutions that did have internal reviews, we found' 
definite signs of expansion. In some cases, concern about the 
quality of the undergraduate programs has encouraged deans, 
faculty, and academic officers to expand graduate reviews to 
lower-level programs. A few institutions, such as the University 
of Iowa, currently include research and service programs in their 
formal reviews as part of an effort to review all institutional acti- 
■vities. Similarly, many institutions show signs of carrying out 
their review procedures with an increased rigor. At institutions 
such as the University of lUinois-Urbana/Champaign, the 
University of Minnesota, and the University of Iowa, such 
changes have come in the wake of evaluations done on the 
review procedures themselves. 

Comparisons with processes of a few years ago indicate that 
reviews are increasingly apt to utilize multiple review indicators, 
which are often plugged into detailed computer data bases. 
Typically, institutions report stronger reviews thanks to improved 
means for gathering program-costing information, as well as use 
of more sophisticated outcome-type measures such as alumni 
and employer surveys. Also, consultants are increasingly 
brought in either from other programs at the same college or 
university .or from similar disciplines at other institutions. 

Rewards accruing to this increased rigor are well documented. 
DiBiasio (1981) has studied in-depth the program-review system 
adopted at Ohio State University (see also Arns and Poland 
1979). In his preliminary findings, he found that improved 
methods provided a better overall perception of the program,' 
which, in turn, helped promote fiscal and governance changes as 
well as better communication with other programs. Also, faculty. 
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curriculum, and equipment resources were all put to morcipro- 
ductivc use. DiBiasio further discovered a number oPso-called 
"secondary consequences." Among them was annncreased faculty 
communication both with one another and with administrators. 
In some cases, DiBiasio found that the review process brought 
out a particular faculty member*s administrative potential in. a 
way that had previously gone untapped. 

Perhaps the most significant impact of recent reviews is their 
growing role in institutional decisionmaking. Originally, pro- 
gram reviews primarily served as information reports, which 
faculty tended to use or discard as chey saw fit. Although many 
of these reviews seemingly did result in substantive improve- 
ments in individual departments, few outside individuals ever 
saw them. Today, not only administrators but many faculty ask 
for wider review distribution. As one faculty report put it, "The 
review documents constitute, when properly done, . . . the single 
most valuable and comprehensive.. source of information 
a\'ailable to the colleges, the Graduate School and central ad- 
ministration in planning for the future^of this university." 

Examples of such expanded distribution can be found at the 
University of Vermont and Ohio State University, where pro- 
gram reviews provide a critical link between planning and 
budgeting. Through a-mechanism known as the "Memorandum 
of Understanding. . .the annual budget becomes less- an 
unspecified grant and involves performance expectations and ex- 
plicit objectives as well as dollars" (Arns and Poland 1979, p. 13). 

In like manner, the University of Louisville circulates program 
reviews among all of the vice-presidents and to a planning staff 
including people in the planning office, budget office, and office 
of institutional research. The University of Louisville has 
recognized that the review process cannot be separated, from 
overall planning. According to a university document, "Neither 
is meaningful without the other. A plan to achieve stated objec- 
tives is not complete until alternatives are considered and their 
respective resource requirements are compared." The following 
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. list includes a few of the many objectives of the university's 
**programmatic planning reviews'^ 

• To involve units in an assessment of their current objectives 
and activities in relation to institutionwide goals 

• To begin the process of collecting output information 
appropriate to each unit 

• To begin the process of identifying, for each unit, the 
resources used (facilities, finances, and personnel) 

. • To provide a basis for recommendations regarding (1) 
' intern.al allocations in the preparation of annual operating 
budgets and (2) reallocations (in the form of budget adjust- 
ments) during the operating year 

• To serve as a building block in an evolving planning pro- 
cess chat will make planning and resource allocation 

. increasingly rational processes 

At other institutions faced with conditions of financial exi- 
gency, program review is increasingly used as an eleventh-hour 
device for setting program priorities (Dougherty 1979). Various 
processes are now available for linking program revievv vvith 
resource allocation decisions. Based on their hands-on experi- 
ence, Shirley and Volkwein (1978, p. 25) suggest using reviews at 
times of financial exigency to "extract programs not meeting, 
(program) priorities . . .[and to] preserve the vitality of those at 
the heart of the intellectual enterprise. '"They further believe 
that "these decisions are best made at the campus level within 
th^- context of an overall , academic planning process which 
involves faculty, students, and administrators.'' . 

Mosf-ldeally, planning, priority setting, and review processes, 
are well established prior to. the onset of financial exigency. To 
those ends, a number of colleges and universities are actively 
implementing planning processes that use program review to set 
priorities among institutional programs. To gain insight into the 
' actual application of internal review procedures, the reader is 
referred to the examples provided in Appendix III. • 
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Basics of Program Review 



Pu.' )ses 

Whcr asked why they decided to involve themselves in internal 
proyn' vn reviews, the surveyed institutions mainly indicated that 
tllcy wanted to improve their academic programs. Many pointed 
out that without knowing a program's strengths and weaknesses, 
such improvements were virtually impossible. Institutional 
literature tends to substantiate this point. For example, a report 
by the Committee on Internal Program Review of the Council of 
Graduate Schools states simply that "the purpose of a well con- 
ducted review process is to help the program improve" (Gentile 
1980, p. 7). Although such statements are no doubt accurate in 
their own right, a common difficulty arises if faculty take "im- 
provement" to mean that resources will be added later on when, 
in fact, they will not. 

Perhaps not so surprisingly, the second most commonly cited, 
reason for conducting internal program reviews is to redistribute 
resources and reduce programs. Understandably^ this second 
re.ason was most frequently cited by institutions undergoing 
severe enrollment declines and retrenchment. q 

Participants 

Because of the need to draw on widely ranging perspectives 
and expertise, the role played by review participants is crucial. In 
each instance, it must be decided who will be involved and to 
what extent they will have responsibility. 

Program Faculty. As might be expected, program -faculty are 
key figures in the review process. Although in limited cases 
departmental faculty conduct all aspects of the review them- 
selves, we found that generally faculty concentrate on data 
gathering and the self-study aspects of the review.' 
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Consultants. To obtain objectivity, most institutions try to in- 
volve faculty from outside the department under review/Outside 
consultants with general backgrounds, as opposed to disciplinary 
backgrounds, are rarely used except to consult on the review pro- 
cess itself. No matter where their home territory is— on campus 
or off campus— these consultants usually have influential roles to 
play either as members of review . teams or as participants on 
institutionwide committees. More than a quarter (26 percent) of 
our survey respondents remarked that campuswide committees 
are the final arbiters in reviewing campus programs. 

By and large, outside peer faculty are hired as consultants, with 
salaries ranging from gratis to upward of $250 a day plus 
expenses. Often, when a lower rate ..of pay is provided, they 
receive an additional stipend for writing the report, an activity 
that is usually estimated to take anywhere from one to four 
mo.nths to complete. Although the time consultants spend on 
campus may vary from one day to about five days, these visits 
typically include interviews with the program faculty, students, 
and administrators, plus appointments at related facilities such 
as laboratories or libraries. Scheduling formalities vary tremen- 
dously, but most consultants are free to-browse and investigate 
as they wish. In a few instances, however, consultants are not 
permitted to socialize with program faculty, particularly when 
internal evaluations are done in conjunction with external 
reviews. ^ . . 

According tO at least some estimates, three is the ideal h;umber 
of peer consultants. As one respondent put it, "The use of three 
consultants enables you to balance the team [ideological differ- 
ences within a discipline], allow for a swing vote, and provide 
more breadth and depth into the particular discipline or field." 
Although only 5 percent of the respondents indicated that the 
use of consultants was mandatory, roughly half of the institu- 
tions conducting reviews chose to use them. This no doubt 
reflects a relatively high rate of satisfaction. 

Of those respondents indicating that they made use of outside 
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peer consultants, 17 percent said that they were "extremely 
satisfied," about 75 percent indicated that they were "somewhat 
satisfied," 7 percent were "somewhat dissatisfied," and only 1 
percent reported they were not satisfied at all. Those who were 
dissatisfied complained that the consultants either "didn't 
understand the local situation" and seemed "biased for or 
against the department under review," or that they were "too 
costly." But those who championed the consultants countered 
that consultants seemed to be "the only way to bring objectivity 
to the review" and that they were "needed to assess the quality 
dimensions of the program." 

Others. Other key participants in the review process include 
deans (usually a school or institutional academic- dean or a 
graduate dean), the vice-president for academic affairs, and 
students. Of these groups, the dean usually selects the con- 
sultants and outside faculty and also organizes and schedules the 
review. Although the academic vice-president may assist in some 
of these activities, he or she is more likely to be the one who 
decides whether or not to integrate the review into the planning 
and budgeting process. Also, if a final decision is to be made on 
the program review, the academic vice-president is the one 56 
percent of the respondents noted as making it. * 

When it comes to student involvement in program review, the 
level of participation depends greatly on the amount of time- 
students are willing and able to devote to the process. Generally, 
students indicated only mild interest in the reviews except where 
controversy had piqued interest in some particular aspect. Some 
students, however, are now starting to serve on institutionwide 
review committees, where they either may or may not have 
voting, privileges.. The significance of student participation 
depends largely on the internal characteristics of the particular 
institution; Exceptions apart, the most frequent role of current 
students, graduates, and former students is as participants in 
surveys. . 
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In cases where the president and trustees become involved in 
the reviews, their roles are often limited but nonetheless critical. 
As one graduate-school committee on internal review put it, 
"Without a clear-cut mandate from the President to conduct a 
program review, administrators at all levels are placed in a dif- 
ficult position. If the President does not see the value and per- 
tinence of program review, the process will quickly become a 
useless appendage . , (Gentile 1980, p. 4). Obviously, most 
reviews depend to one degree or another on the support of the 
■president and trustees, even when their actual participation is 
minimal. 

Costs 

, Not counting the time contributed by faculty, staff, and 
administrators, the average cost per program for conducting an 
internal program review is around $4,000 to $6,000, of which the j 
largest portion goes for hiring outside peer consultants. Since the 
expenditures are generally borne by the institution itself, it is not 
surprising to find that opinions vary considerably as to whether 
or not the bill is worth it., Although some consider the costs 
necessary to ensure program quality and efficiency, others view 
such expenses as justification for either not reviewing programs 
or limiting the scope and frequency of the reviews. 

Selecting the Program 

Typically programs are reviewed when their turn comes up on 
a review cycle for all departments. Others, however, are flagged 
either by a monitoring process, an audit, or a screening method." 
For some institutions this flagging may be done by the state 
agency for higher education, which is most often concerned 
about whether the program in question has attracted arid 
graduated enough students over a specified time period. In these 
. cases, program productivity—measured in terms of enrollment 



42 

48 



REVIEWING PROGRAMS INTERNALLY 



and number of graduates— ranked first of the criteria used for 
flagging programs, followed by program costs, job scarcity for 
graduates, and general funding limits. 

Even many of the institutions that rely on the cyclical pattern 
employ these criteria to begin their cycles. Some institutions also 
indicated that unusual circumstances such as an enrollment 
drop, a negative accreditation report, or a change in administra- 
tive staff is considered just cause for triggering an off-cycle 
review. To illustrate. the last point, the University of Michigan 
ordinarily reviews an entire colfegc whenever a new dean is ap- 
pointed. Tl*i^ review not only gives the new dean a better overall 
perspective but also provides viable information for reevaluating 
the strengths and weaknesses of all programs in the college. 
When it comes to the bottom line-, our survey indicates that 
when p: arams are selected for review considerable flexibility is 
employed by administrators at most colleges and universities. 
Unusual circumstances in a department, such as the sabbatical 
leave of a key faculty member or a pending professional accredi- 
tation visit, can easily alter review dates. . 

Almost'three-fourths of theinstitutions surveyed try to com- 
bine their internal reviews with professional accreditation visitSi 
in an attempt to conserve energy and improve reviews. Some 
program evaluators, however, think reviews are not improved, 
saying that the two evaluations are at cross-purposes. In their 
view, program evaluation can generally be more objective in. 
terms of resource allocation when it's separated frou'. profes- 
sional accreditation visits. As one academic vice-president put it, 
"Harcily an accreditation visit goes by that doesn*t recommend 
additional travel funds, more or better equipment," higher 
salaries, lower workloads, et cetera.** To avoid possible conflicts 
of interest, some, accrediting agencies forbid joint reviews. 

Selecting the Review Criteria 

The single most difficult and time-consuming aspect of pro-' 
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..gram reyievv^^^^^ is selecting appropriate review criteria. 

Although such choices are difficult in any setting, the'probrernTs~" 
compounded at the college or universitywide level simply 
because of the variety of programs and program levels, such as 
undergraduate, professional, master *s, doctoral, and so on. 
Sparks (1980) suggests that master's programs alone can be fur- 
ther divided into four major categories: (1) traditional academic 
programs, (2) traditional professional programs, (3) specialized 
training programs, and (4) nontraditional programs. 

Of the criteria used for program review, 'Equality** is certainly 
the most elusive, as well as the most controversial. As Blackburn 
and Lingenfelter • (1973) explained almost a decade ago: 

A desire for highest quality seemingly needs no defense, irre- 
spective of the product or craft involved. Yet, the considera- 
tion of "quality" in any setting seems to provoke controversy. 
Whenever evaluative judgments are made and a rank order is 
established, someone is offended. Hence, the evaluator, 
sometimes correctly, sometimes in error, is likely to be charged 
with elitism, bias, racism, or a host of other unsavory traits. 

No doubt much of the controversy about quality relates to 
ambiguities within the concept itself. After all, as Cartter noted, 
'In an operational sense, quality is someone's subjective assess- 
ment, for there is no way of objectively measuring what is in 
essence an attribute of value" (1973, p. ii). Nonetheless, most 
reviewers finally agree on some approach to reward quality 
assessment, as well as quantity. Figure 1 1 shows the criteria used 
in reviewing programs, broken down according lio the stated 
needs for the program, the costs and benefits, and other related 
objectives. 
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Benefits 

Although most survey respondents seemed satisfied with their 
preseiv review efforts, a fairly large number of institutions (35 
percent' indicated that further refinements were needed. The 
most frequently mentioned target areas included program data 
collection and integration of review reports into budgeting and 
plajnning procedures. 

Despite such obvious n^eds, however, the list of benefits derived 
from program review is long and varied. For the most part the 
benefits are consistent with those noted by Sparks (1980, p. 8), 
who mentions the following: 

• An increased attention to our responsibilities to our 
students 

• Increased consideration of alternative ways to develop and 
deliver . . .programs 

• A more certain basis for interinstitutional, as>yell as inter- 
departmental, corriparisons of content and quality 

• Improved capacity for planning and for decisions on the 
allocation of resources 

• Earlier warning of impending difficulties in "a particular 
program 

I. ' 

Results and Recommendations 

When the reviews are completed, an internal draft report is 
usually circulated, summarizing the strengths as well as 
weaknesses of the particular program. At that time, program 
faculty may respond to negative or positive commentaries. 
Often, a negotiation process then takes place involving academic 
administrators as mediators between the reviewers and those 
reviewed. 

By and large, the vast majority of the recommendations coming' 
out of institutional reviews focus on program improvement. 
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Mostly, they addrcss-concerns about program faculty and sug- 
gest ways to improve the progrnm*s operation. Although some 
recommendations may indicate a need for added resources (such 
as equipment, faculty, facilities, salaries, travel to professional 
meetings, and so on); such recommendations are less likely when 
the review's primary focus is larger than the immediate program. 
Then recommendations may suggest resource reduction or even 
program termination. ' - 

Some evidence, however, indicates that resource reallocation 
can positively affect an institution. Guardo (1980, p. 8), for 
example, reported that at one institution the reallocation pro- 
cess resulted in "heightened morale as a consequence of the suc- 
cess of curricular initiatives and optimism about the ability to 
respond to changing conditions in the future.'' 

If closure happens to be the review outcome, discretion is of the 
utmost necessity. For obvious reasons, program closure can be 
traumatic. At least one. writer has likened the situation to the 
circumstances surrounding death, suggesting that aspects of the 
grieving process, such as denial^ anger, bargaining, and depres- 
sion, should be addressed by administrators to help faculty and 
studentii cope with the impending termination (Davis 1980). 
When the evidence has been objectively gathered 'and fairly 
presented, a few cases exist where a program faculty has actually 
recommended its own termination. 



CHAPTER 4 



Reviewing 
Programs at 
Systein and 
State Levels 



Systenri'Lt^vel Reviews 



In many ways, program reviews at the system level are not , 
that different from* either institutional or state^level reviews 
(Rudnick 1976; Srhith 1980). In fact, they generally combine 
elements of both. As one might expect, with a smaller system or 
district the review will probably resemble an institutional 
review, and^the larger the system the greater the likelihood that 
it will resemble a state-level process. In other words, smaller 
systems tend to be more formative and Jarger systems more 
summativc. - 

Approximately h^lf of the system-level office^ for both four- 
year institutions and community colleges undertake some level 
of program review. Although some system offices limit their 
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reviews to simply monitoring either stacc'level or institutional 
reviews, more comprehensive system-level reviews appear to be 
rapidly increasing. Tighter economic conditions, declining 
enrollments, and demands for more efficient resource manage- 
ment have combined to bring system-level operations to the 
front lines of reviewing efforts. In California, for example, 
where the California Commission on Postsecondary Education 
limits its role to monitoring the quality of the system-level 
review efforts, the reviews have become second only to those 
performed by the institutions. 

Along with this growth, however, have come frequent con- 
frontations and jurisdictional disputes when system offices find 
themselves caught between the institutions and the state agen- 
cies. Some of the system offices visited felt uneasy about their 
ambiguous role. This was particularly so in cases where the 
system offices felt ignored^br bypassed in the review process. For 
example, one system office staff that played a large budgetary 
role complained that it didn't even get courtesy copies of the 
institutions' program reviews from the state coordinating 
board. In another system, the central office staff explained that 
they were forced to set up their own review processes after being 
ignored by the state coordinating board. As a result, that state 
now has three separate reviews and review processes serving the 
constituent institutions. The role-conflict problem is even fur- 
ther exacerbated when the system-! '"vel office assumes a major 
responsibility for resource allocation. Then attempts at mean- 
ingful use of the review for resource allocation are frequently 
either not possible or thwarted by jurisdictional disputes with 
the state coordinating agency. 

To date, there doesn't appear to be any direct correlation 
between the level of program-review activity and the size of the 
system. We found that several large systems had no role— or'no 
meaningful role — while some small systehas were heavily in- 
volved in program review. Generally speaking, the presence or 
absence of reviews from the state coordinating board bore little 
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relationship to the level of system-review activity. In some 
states, both the coordinating board and the system office 
reviewed prograriis, whereas in other states, the system office 
avoided conducting separate reviews altogether. 

Because of the similarity in.actual content between system and 
state-level reviews, we will now move our discussion to the latter 
categoiy, but before doing so, there is one final point to make 
regarding the general influence of system-level reviews. In some 
instances, actions are directly taken by the system board or its 
staff In others, the reviews are used more as a means of pressur- 
ing the institutions to take corrective action. The exact approach 
used by a system seemingly depends more on tradition than on 
legal authority. Exceptions apart, systems that primarily employ 
decentralized decisionmaking tend to leave the end results up to 
the institution, whereas those with a more centralized tradition 
are apt to exert a stronger influence. (See Smith 1980 and Craven 
1980 for further discussion of system reviews.) 



State-Level Reviews 



" Just as system-level offices have come to assume larger-responsi- 
bility for program review, so too have the state agencies. In fact,- 
program review has consistently ranked in the top ten "Major 
Issues of Concern to State Higher Education Agencies'' (Millard 
1977; 1978; 1979; 1980; Berve 1981). Explanation for such consi- 
derable interest is found in the words of former Indiana governor 
Otis R. Bowen, who noted that higher education's growth came 
without adequate preparation and without adequate question- 
ing. As Bowen explained the situation in his home state in 1979, 
"We have nearly 3,000 degree programs in Indiana and only two 
staff people to monitor them. This means institutions must do 
the major job of program review. Yet institutional administra- 
tors have been the least willing* to make such hard decisions. 
Statewide boards must put into place processes which demand 
and support institutional attention to this area." 
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By latest count, all 50 states have some sort of reviewing pro- 
cess, although specific responsibilities vary greatly from state to 
state (see figure 12). According to our survey, 23 of the SHEEO 
boards conducting program reviews are governing-board types. 
Out of these, 11 are comprehensive boards (governing most or all 
of the public postsecondary institutions), and 12 govern only the 
senior public institutions. Besides these, 20 states offer coordin- 
ating boards and 41 states have planning commissions. 

FIGURE 12 

Status of State-Level Higher Education Agency Program Reviews 

(Review of Existing Programs 




I I State conductinR program review 

[■"Vj State developing progrnm review process 

[Ph Review of selected programs only 



Cooperative review between state and institution 

(SOURCE: Robert j. Barak, State-level HtKhcr Eduauiim PmRram Rcvk-w U/jc/(Uc: 1979. 
SHI-EO/N'CES Comnumieations Network, 1979,) 
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Figure 13 further explains the various review responsibilities 
by indicating the state-level agencies in termitof their authority. 
Similar findings can be found in Melchiori (1980), Education 
Commission of the States (1980), Skubal (1979), and Wilson 
(1980). 

FIGURE 13 

Authority to Review at Least Some 
Existing Programs by Governance Type 
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Tahulntions such as these are usefuh However^ only to a certain 
point. Beyonci them^ additional interpretation is required. For 
example^ some states exercise authority above and beyond what 
their statutory or constitutional authority would suggest^ and 
others^ of course, exercise less. Some agencies may, in fact, repre- 
sent only the public sector (or even just a portion of the public 
sector), and others may have authority only for undergraduate 
or graduate programs. , 

All told, 21 of the 23 governing boards have authority to 
discontinue programs, while just 2 have authority only to review 
and recommend. Seven of the regulatory coordinating boards 
can .discontinue programs, but, admittedly, this authority is 
sometimes limited to just one institutional segment. Eleven of 
these boards, however, can review and recommend. As far as the 
advisory coordinating boards ave concerned, none has authority 
to discontinue programs, 5 can review and recommer.d, and 5 
others have absolutely no review authority. However, some 
boards with authority for program review hardly exercise it, and 
at least 1 agency —presumed to'be without authority— has suc- 
cessfully reviewed and achieved terminations of some graduate 
programs. Such discrepancies usually relate directly to political 
environments, traditions, and agency capabilities. (For a com- 
prehensive background on the state boards, see Millard 1976.) 

Reasons for condu 'ting reviews may vary from a desire to im- 
prove quality or modernize programs to efforts to encourage 
better institutional planning and establish consistency with 
institutional and state missions. Furthermore, a recent study 
(Barak and E6gdahl 1980, p. 128)ofstate-level program reviewin 
the Western Interstate Commission for Higher Education states 
revealed the following additional purposes: 

Maintenance of articulation and coordination among 
parts of a complex system of postsecondary education 
• Facilitation of planning for postsecondary education in 
the state 
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• Elimination of unnecessary spending couscd by program, 
duplication 

• Assurance that available resources are adequate, for quality 
' educatioii 

• Identification of programs that do not meet minimum 
criteria as a basis for deciding wliethe'* to f,/iiminate or 
strengthen them 



State-Level Approaclies 



Basically, state-level reviews are conducted according to two 
general approaches. First, state agencies can share the responsi- 
bility for review with their constituent institutions by providing 
encouragement, perspective, and monitoring. And second, state 
agencies may assume the major responsibility for reviewing 
existing academic programs. 

Agencies That Share Responsibilities 

About a dozen state-level agencies share review responsibilities 
V ith colleges and universities. While the exact form may vary, 
and despite the fact that the specific program for review may be 
selected by the state agency,, review responsibility rests largely 
with the particular institu\ion. For its part] the state agency 
generally provides special or lateral review of the same discipline 
or clusters of disciplines across all institutions. Stated agency 
■ reasons for taking this approach vary from practical considera- 
tions such as having a lack either of authority or of agency 
i^. sources, to a belief that educational institutions themselves are 
best suited for judging their pro!_;rams and that institutional 
autonomy must be protected. Some states that use this general 
approach include Illinois, Idaho, New Mexico, California, 
Oregon, and Ohio. 
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Illinois offers an interesting example of the sharing approach, if 
for no other reason because it cnme to adopt this philosophy 
only after the publication of a new master plan in 1976, which 
clearly identified a role for the Illinois Board of Higher Education 
(IBHE). Prior t(^ this, despite a fairly explicit mandate in the 
agency's authorizing statute, IBHE demonstrated little interest 
in program review. 

Today, Illinois utilizes two different processes for reviewing 
public university programs: institutional reviews and statewide 
lateral or generic reviews. At the institutional level, approxi- 
mately one-fifth of the programs within each institution are 
j-eviewed yearly thanks to a five-year schedule developed at each 
public Liiiiversity and community college. And in turn, these 
reviews are complemented at the statewide level when all degree 
programs within a discipline or set of similar disciplines are 
reviewed within a given year. Statewide reviews focus primarily 
upon programmatic policy and planning issues within the pro- 
gram area as opposed to more individualized treatment at the 
institutional level. 

The way the inciividual treatment works is that each year the 
Illinois institutions provicie IBHE with two-page recommenda- 
tion summaries plus a rationale for each program reviewed dur- 
ing the preceding year. The IBHE staff then consults with system 
and campus representatives in developing recommendations 
about economic and educational justifications for the programs 
reviewed- This process culminates in an annual report to IBHE 
on each institution-level review. 

Then, when the statewide reviews are done, IBHE frequently 
expands the scope to include participation by the private sector 
as well. Roderick T. Groves,' who serves as deputy director of 
academic affairs of the Illinois Board of Regents has made the 
following comments about the Illinois approach: 

Such a division of labor offers a number of advantages. First, 
the system if properly developed will provide for, the regular 
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review o( iiciidcmic prt^grnms in n fashion that is consistent 
with Lliffcrcnt institutional needs and is relatively conserving 
of time and efft^rt because of flexibility and reliance oivdecen- 
iralized initiative. Second, it holds a potential for strengthen- 
ing the efficacy t)f the existing adniinistrative structure of 
Illinois higher education because it will work in a manner con- 
sistent with that structure rather than at cross-purposes. 
Finally, and most important, by pulling the existing university 
and gt)verning bt)ard structures of program review under o 
statewide umbrella, it can considerably increase thelegitimacy 
of those efft)rts. 

Regardless of internal perceptions and biases, higher educa- 
tion is generally viewed from the outside as a single sector of 
statewide activity. In all probability external respect for a sup- 
port of higher education's efforts at program review will be 
even more important in the future than at present, so legiti- 
macy is an important consideration. (Groves 1979, p. 20) 

Other states that share reviewing responsibility range from 
California (where, as previously mentioned, the Postsecondnry 
Education Commission limits its role exclusively to encourage- 
ment and review at the system level) to states at the opposite end 
of the spectrum. For example, in Oklahoma the State Board of 
Regents utilizes a. process that ties program planning and fiscal 
budgeting together in such a way that no new educational pro- 
gram is approved without money first being placed in the budget 
for its implementation. Since no new program is initiated until 
all existing programs have been funded at the 100 percent level, 
institutions are provided considerable incentive for deleting old 
programs in favor of new ones. 

The issue of institutional involvement is obviously a complex 
one, regardless of whether it s done in a joint'review or exclu- 
sively by the state agency. In any case, a critical credibility factor 
relates to the degree and meaningfulness of institutional involve- 
ment in the process (Barak and Bcrdahi 1978). In this regard, a 
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sUilIv hy Richaril P. Wilson notes four possible alternntives for 
institiuional participation in state-level program-review activi- 
ties, each of which provides a unique access to decisionmaking^: 
(1) reactive, (2) advisory, (3) formative, and (4) voting (Wilson 
1980), Figure 14, taken from Wilson s study , briefly defines these 
four approaches. 

FIGURE 14 
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According to Wilson, each of the strategies provides institu- 
tions with a "different access to decisionmaking and has direct 
implications for the credibility of review results" (p. 20). From an 
institutional perspective, formative participation is especially 
attractive because it affords possible involvement in virtually all 
of the- review stages. Like other aspects of the review process, 
however, the specific institutional role is largely determined by a 
variety of factors, including politics and personalities, statutory 
authority, and tradition. 
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Agencies That Assume Mnjt)r llcsponsihility 

In soinc 30 states, state n^eiicies assume the major respoiisihility 
themselves for program review at puhlic institutions. And in 9 of 
these states, they eonduet at least some reviews of private eollegcs 
and universities as well. Many in the latter eategory are ineluded 
on a voluntary basis. To one degree or another, most state agen- 
eies that perform eomprehensive program reviews rely on a 
eonibination of outside eonsultants and agency staff. To better 
understand the consultant-based approach, we'll first look at the 
procedures iis applied in Louisiana. 

r V 

Consultant-based Approach. .Confronted with concerns 
about doctoral-program duplication, the Louisiana Board of 
Regents undertook a series of reviews in 1975 modeled after a 
similar application in New York. The revievv'S originally began by 
examining tlx; doctoral programs thought to be duplications, 
and later expanded to other program areas. 

Today, Louisiana's review process begins with self-studies, 
which are designed to acquire data on aspects such as curriculum, 
finances, library resources, and so on. Outside peer consultants 
are then selected from lists prepared by the institution and from 
recommendations from learned societies. Upon completion of 
one-day visits and an analysis of the self-studies, the consultants 
prepare qualitative evaluations., Following public hearings on 
those'evaluatlons, the regent staff then makes recommendations 
to the board. Using this procedure, 76 separate programs had 
been reviewed by 1978. Of these, 20 programs were terminated, 
48 programs were identified to be maintained and strengthened, 
and 8 programs were awarded commendations of excellence. 

In reviewing doctoral programs, the Louisiana Board of 
Regents has adopted a set of considerations that, among other 
points, encourages institutions to regularly review their Ph.D. 
programs while reviewing "all the state*s doctoral programs as 
constituting an interrelated system for doctoral education.** 
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Agency Staff Approacli. A second general approach to state- 
level reviews relies almost exclusively on agency staff rather than 
consultants. Generally speaking, states using this approach 
depend heavily on data provideel by the institutions iind/or the 
agency. 

The Council for Postsecondary Education (CPE) in the state of 
Washington conducts reviews of its doctoral programs in this 
manner. The CPE reviews have taken place in different phases. 
The first phase, known as GPA-I, screened out programs that 
indicated chronic signs of low productivity. 

Phase two, called GPA-II, focused on program duplication in 
either the same field or degree level at public institutions. In 
undertaking the second phase, graduate deans were consulted 
regarding concerns they had about the mechanics of the audit 
and review process, as well as any additional problems relating to 
data collection. Further program information was then acquired 
via c|uestionnaircs, which were designed by both the deans and 
the staff. When this procedure was first begun, the quantitative 
data was converted to machine-readable form and a computer 
analysis was made. But when it soon became apparent that many 
subtleties were being lost, this approach was dropped in favor of 
one using only manual calculations. 

In any case, when no additional information is required for the 
CPE reviews, preliminary staff reports are presented to the coun- 
cil as well as to the faculty of adversely affected programs. The 
CPE staff then visits each institution to meet with 'faculty, 
administrators, and students to present recommendations, hear 
defenses, and discuss alternatives. Finai staff recommendations 
are then made to the CPE. 



Presenting the Evidence 



At one point or another, the findings of the program reviews 
must be made public. But when, where, and how this is handled 
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varies 'grcjrly from state to state: In New York, consultant 
reports are regarded as strictly advisc^iry; they are not nuide 
public except at the request of the particular institution. Before 
any official announcement is made, a rating team consolidates 
the individual consultant reports from all of the programs in a 
particular discipline. And these reports, while still not public, are 
read by a doctoral council, at which time all institutions under 
review are represented. The final decision on the reviews is made 
by the commissioner according to>the rating-committee report 
and recommendations. The decision is made independently of 
the New York Board of Regents. 

lu Louisiana, on the other hand, the staff bases its recommen- 
dations on consultant reports, which are then presented at 
public hearings before a subcommittee of the board of regents. 
Because the hearings focus on public issues and report content, 
public institutions must respond to any weakness noted in the 
report, whereas the private institutions are merely invited to 
respond. Following the hearings, the board of regents makes its 
final decisions on the program review. 

Both of these examples show approaches used in developing 
and making public the program-review reports. While the proce- 
dures and practices vary among the states, the meeting at which 
the state agency 'icly discusses its findings can be highly emo- 
tional, particuli t negative results are reported or program 
termination is recommended. In fact, some agencies refuse to 
make final decisions until one meeting later just to reduce the 
impart of the turmoil. 

Beyond the agency findings, further controversy may arise 
from concern about public disclosure of the actual numbers of 
programs terminated. Some agencies try to play down the figures, 
whereas others distribute the results nationally. Whatever the 
approach , specific tallies of terminated programs tend to bear little 
relationship to agency effectiveness. For example, one state- 
carefully monitored resource allocation over the years and 
thereby prevented most weak programs from developing. In a 
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iKsirhy sintc, progriiin prolifLM-ntitm niul cluplicaiiou wcru 
nuupiUU . C Aiusciiuciuly, the iniliiil reviews showed only modest 
pro^riini lemiiniitions in the first stiue while the neiirhy stnte 
tenuiiuueJ liiernlly hundreds of prti^rnms. 

Whether nv not etintroversy arises, whei'i the review process is 
eoi^ipleted w deeisit)n must he made regarding the actunl steps 
necessary for the program to undertake. Roughly half of the 
rt;j<*iewing agencies only make recommendations, either to the 
institution's governing board, ^'^J^^^' system office, or to the 
governor ov legislature, The otheV hfilf actually initiates formal 
action. According to an exhavistiv^e study of stntc'lcvel program 
disctmtinuance (Melchiori 1980), t^ie range of options utilized by 
agencies with formal power'may inctffdq decisions to continue, 
modify, merge, or terminate programs. '"'^^ 

Approximately 14 state agencies in our present study indicated 
that programs had been discontinued as a consequence of their 
review activities. Similar findings were reported by Skubal 
(1979), where 12 agencies, or 71 percent of the agencies report- 
ing, indicated that programs had been discontinued since 1970. 

According to Melchiori (1980), four levels of program termina- 
tion can be identified: (1) the elimination of "paper programs'* 
that arc often maintained by academic departments for flexibility; 
(2) the elimination of programs with no adverse consequences for 
students and no release of tenured faculty; (3) the elimination of 
programs with serious consequences where tenured faculty and 
students are adversely affected; and (4) the elimination of entire 
units, departments, or collegcs.if we apply these standards to our 
present study, we find the following percentage breakdowns: 



Level of Action 



Percent 



Level One 
Level Two 
Level Three 
Level Four 



30 
61 
8 



100% 
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More than anything else, these tnbuhicions indicate a shift in 
recoiu years from Level One to Level Two-type actions. This is 
especially evident in states with lonjjjer histories of program- 
review activity, Recent reviews appear to be conducted under 
greater pressures for accountability and consequently rei|uire 
more thorough resource and personnel adjustments, Although 
two-thirds of the program discontinuance to date has occurred 
with respect to graduate-level progranis (mostly master/s and 
specialist degrees), many state reviews on the undergraduate 
levels appear to be just now getting underway. 



Evaluation of State-Level Reviews 



So far, few attempts have been made to evaluate the evalua- 
tors. Among the exceptions are Alabama and South Carolina, 
where program-review activity has been analyzed as part of a 
comprehensive evaluation of the state agencies. In Alabama, an 
evaluation of the Alabama Commission on Higher Education by 
an external team resulted in a recommendation for expanded 
responsibilities (Alabama Commission on Higher Education, 
March 1979), Similar recommendations were made in South 
Carolina as a result of a. legislative audit (South Carolina G.A,, 
1978), 

Although official reviews have been scarce, unofficial 
reviews— by members of state legislatures as well as colleges and 
universities— have become relatively common. Some suggestions 
from the latter group include focusing reviews more closely on 
academic departments (instead of on programs and degree majors, 
which may be only loosely connected with the administrative 
structure) and zeroing in on ways to maintain efficiency and ex- 
cellence (instead pf randomly searching for an institution's 
weakest programs). Some critics even stress that the work may 
have been largely completed in eliminating weak programs and 
that it's now time to dwell on more supportive aspects of review. 
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Bogue (1980, pp. 81-83), for example, has described a program 
evaluation as a "Renewal Instrument" that can be employed in a 
variety of ways. Some of the renewal functions include clarifying 
program goals, expanding understanding of quality, and 
strengtht*ning community and personnel development. 

The present study has added to this list the following benefits 
suggested by individuals involved in state-level reviews: 

1. Improves resource allocation 

2. Helps in making difficult program decisions on a statewide 
basis 

3. Makes termination of programs less hectic at institutional 
levels since some external body is blamed, even though the 
institution's administration concurs (at least privately) 
with the decision 

4. Tightens admission standards - 

5. Clarifies institutional missions 

6. Encourages interinstitutional cooperation 

7. Promotes better institutional management 

8. 'Provides.an acceptable rationale for pruning the system on 
a statewide basis 

While both criticisms and benefits tend to reflect strongly on 
the respondent's individual circumstances, the combined obser- 
vatio'^ns are highly indicative of the various opinions currently 
influencing program review. 
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Program Review 
Systems: Making 
Them Work 



Who has the best program-review system and how can we 
implement it? For evaluators, this becomes a familiar 
question but one with no easy answers. On the basis of this 
study, and the author *s experience as to what works and what 
doesn^t, there doesn't appear to be any best system. There are 
just components of existing systems that may or may not work 
well for a given institution or agency. 

Attempts to plug in an existing system without modifying it to 
the local environment are generally unsuccessful. As Mims 
(1978, p. 4) has written, it's better to "adapt*' than to **adopt** the 
external model. And in a similar vein, Sudweeks and Diamond ■ 
^(1980, p. lA) note that "no two evaluation designs will be the 
same. In each instance the evaluation must be structured t6 serve . 
.the information needs of those involved in the decision-making 
■ process.'' Above all, participants must.be meaningfully involved 
in the review process, and, accordingly, the reviews themselves 
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must adapt to local circumstances. 

The following principles and procedures are provided to help 
with the adapting process. Although program-review develop' 
mem is a highly complex and individualized matter, the univer- 
sality of the key components lielps make them applicable to 
virtually all institutions, system offices, and state-level agencies. 



General Principles of Good Review Practice 



Fairness ' . 

Unless procedures and policies are perceived as fair by both re- 
viewers and reviewees, credibility will be drastically undermined 
(Gentile 1980 and Sparks 1980). Because of this, care must be 
taken to build a review system that treats all programs and insti- 
tutions in a just manner including, ofcourse^ due-process proce- 
dures to resolve disagreements' (Berdahl 1975). Objections 
regarding either review development or implementation should 
be provided fair hearing. 

Comprehensiveness 

Ideally, a program-review system should be comprehensive. 
Not only should it include all relevant programs, but also, to the . 
extent possible, the individual reviews should include all factors 
relating to any particular program. For example, if a program has 
a cooperative education component, that component should be 
evaluated. Too often quality is compromised by haste. Opinions 
of consumers, faculty, peers, and alumni should be collected. 

Closely integrated programs and programs that share mutual 
concerns should be reviewed together. Similarly, consideration 
should be given to a review of related curricular areas, such as 
-liberal-arts programs that make up a core curriculum. 
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Use of Malti'Cviteria 

To be effective, programs should be nRasured against mqre 
than just one criterion (Lawrence and Green 1980; Clark 1977; 
1979). For example, factors such as cost, quality, need, and cen- 
trality to-state or institutional mission should be evaluated along 
with several indicators of each of these criteria. 

Both the criteria and their indicators should then be generated 
and assessed by multiple sources, such as students, faculty, 
alumni, and peers. This not only ensi a balanced qualitative 
revievv but also helps to identify problem areas from a variety of 
perspectives. 

Cyclical and Timely Reviews 

Reviews provided on a regular basis (such as five- or six-year 
cycles) are generally more effective than one-shot procedures'. In 
fact, even reviews done every ten years in conjunction with 
regional accreditation visits are usually found to be lacking. Par- 
ticularly in times of economic uncertainty, faculty, .resources,, 
and other, program aspects change too rapidly to risk reviewing 
them just once a decade. Some institutions and states rioW do 
supplemental monitoring or auditing on a yearly basis to identify 
possible problem areas ahead of time. As one graduate school 
report notes, "Through repetitive review, the results of earlier 
actions taken to improve programs can also be assessed, an ex- 
tremely important input to academic management^' (Gentile 
1980, p. 4). 

Good Communications 

Because last minute surprises often cause confusion and affect 
the overall review quality, good.communication is essential at all 
levels. Frequently referred to as the "climate component,'' viable 
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communications should address the ^^development and mainte- 
nance of information dissemination to provide as clear a picture 
as possible of the goals, objectives, and major purposes of the 
review and evaluation" (University of North Carolina 1977, 
Report 2, pp. 6-7). 

A direct correlation emerges between effective communication 
and the development of positive attitudes on the part of the indi- 
viduals whose programs are being reviewed. The more open the 
interaction between the reviewers, the reviewees, and the decision- 
makers, the better the chances of a fair, impartial, and effective 
review. 

Positive Emphasis 

Without avoiding negative aspects— such as possible resource 
loss or program termination— emphasis should be placed on 
positive review aspects. Somehow a balance needs to be struck. 
Overly pessimistic communications can cause morale problems, 
while overly optimistic communications may foster false hopes. 
In short, be positive but realistic. Remember, negative implica- 
tions can be offset by approaches that reward excellence as well. 
Either conceivable resource reallocation or special commenda- 
tions can help provide incentive to those being reviewed. ■ 

Proper Implementation 

Nothing is more frustrating to faculty than to go through an 
extensive review process and only later . to find out that the 
review is not being used for effective decisionmaking. Thus, 
serious efforts should be made to incorporate reviews into the 
appropriate decisionmaking.and budgeting processes. Whenever 
possible, results should be used for program improvement.. 
Although, some institutions try to get by with making only pass- 
ing reference" to review results in official policies, implementation 
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should go beyond just words. As Petrie (1980, p. 1) has noted, 
program reviews "ought to interrelate to other segments or 
systems." 

Objectivity 

Objectivity relates directly to the rigor of the review process. 
For example, if the review purpose is strictly internal then faculty 
self-assessment is probably sufficient. On the other hand, if the 
review's purposes include reallocation of resources or possible 
program termination, then steps need to be taken to introduce 
an equivalent degree of objectivity. 

Our survey results indicate that the degree of objectivity 
relates directly to the credibility of the reviews, especially from 
the perspective of external constituencies. Reviews lackingobjec- 
tivity often result in self-serving reports that contribute neither 
to improved quality nor tq effective decisionmaking. 



How to Develop a Program-Review System 



Certain basic steps have been identified in this study for 
developing or revising a program-review system. Generally 
speaking, they apply at the institutional level as well as system 
and state levels. A flowchart identifying these steps .is found in 
figure 15. 

Because the approach taken here largely resembles a goal- 
based evaluation, factors that are irrelevant for a particular insti- 
tution or agency should be disregarded. 

Other approaches that may be useful include judgmental 
models (which emphasize extrinsic criteria) and decision- 
facilitation models. A general overview of these models can be 
found in Popham (1975), and a more detailed description is con- 
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FIGURE 15 
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tained in Worthen and Sanders (1973). Although most alter- 
native models have not been widely used in higher education^ 
interest has been generated at other educational levels. • 

To reiterate, program review can be overdone. In fact, as we saw 
more than once in this study, it may even stir up other problems, 
resulting not only in a poor review but in the total disruption of 
the institution. Hopefully, the following steps are developed in 
such a way that they may be related to both the review purposes 
and the- users* needs in a functional and harmonious ma '^r. 

Step One— Identifying Purposes and Objectives 

Often called a "needs assessment/' this step helps identify the 
purposes and objectives of the review. Why is the review being 
undertaken? Is it the result of an enrollment decline or retrench- 
ment? Or does it. represent an attempt to improve academic 
quality? Is the review self-imposed or externally imposed? These 
and other questions need to be explored to identify purposes and 
objectives of the review. ■ 
■ Heydinger (1978) suggests that a needs assessment can help 
determine whether or not a program review is even justified. If a 
review is needed, Mims (1978) suggests several groups (adminis- 
. trators, faculty , consultants) who may prove helpful in establish- 
ing and designing a program-review system. Also, Mims' 



^Of further interest to readers interested in additional evaluntiiin models are two documents 
prepared hy consultants to the Ohio Board of Regents {Iiducarion and Economic Systems, Inc. 
l^>7^); Academy for Bdiicational Development l^^^l^ These present a much broader \'iew than 
simply that of Ohii^ instituiions. Similarly, Guha and Lincoln (1981) and Anderson and Ball 
(l^^75) provide valuable perspectives in thejr writings on evakiatiiMi mi^dels and procedures. In 
addition, a host of individual program-review development models are available in Petrie (1980), 
Cranton and Legge (1978), and Heydinger (1977). Useful guidelitu-s for selecting an evaluation 
model and formulating evaluation questions are found in Wood and Davis (1978) and Craven 
(1980). Baderston (1974) provides a set of guidelines that are especially useful for reviev.'ing under 
retrenchment conditions (also see Brown, 1970). And, asa final resource m)te, Heydinger (1978) 
poses the tjuestion of whether or not a particular institution actually needs program review. His 
arguments may help some institutions in deciding if arid when they want to undertake reviews. 
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elaborates on the essential characteristics of the design process as 
a whole. 

These various models and approaches can help the reader gain 
an overview of available options. Since most successful evalua- 
tion models arc eclectic, however, Mims (1978) makes the impor- 
tant point that the chances of setting up an auspicious review 
process are improved if it is user-oriented ficm the beginning. 

Step Two— Conceptualizing the Process 

Assuming that the purpose has been identified either by man- 
cinte or as the result of a needs assessment, the second step is to 
conceptualize the program-review process. At this point, key 
components must be identified by deciding what will be reviewed, 
whm the review will take place, ii;/io .will conduct the review, and 
how it will be dOne. All of these issues need to be considered 
within the organizational context in which they will be imple- 
mented (Patton 1978). 

Similarly, responses to these elements should be uniquely 
related to the review purposes and objectives as identified in step 
one. For example, the reader . may wish to review the model 
developed by Martorana and Kuhns (1977) if the purpose of the 
review is, to identify program duplication in state institutions. In 
another vein, Wilson (1980) suggests that. by setting up a needs- 
assessment study team it*s possible to avoid setting up a dysfunc- 
tional program-evaluation system. The reader may turn to Mims 
(1978) for discussions on designing a program-review system or 
even take the time to explore several models such as those of 
Arns and Poland (1979) and Russo and others (1977). A further 
recourse may be found in the services of program review and 
evaluation specialists. Before a contract is signed, however, care 
should be taken to engage the services of individuals who can ap- 
ply their theories to real life situations. 

Find out exactly what aspects will most desirably fall under the 
review prerogatives. Will all academic programs at the institu- 
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tion, system, or state level be reviewed or will the reviews be 
limited to just certain kinds of programs? Is it possible to cluster 
reviews by discipline or field of study? And how will the reviews 
begin? Will rhey start with pilot tests or will they be phased in 
gradually? Moreover, what definition of program will be used? 
Will program be taken to mean a department, a degree program, 
or perhaps some other category altogether? 

After identification of what programs will be reviewed, some 
determination needs to be made as to it^fien the programs will be 
reviewed. Will pre:)grams be identified by a program audit or an 
ongoing monitoring system , or will they be reviewed on a cyclical 
basis? 

The ciuestion of who will do the review relates directly to the 
general purposes of the review. Mims (1978) identifies three 
distinct approaches: self-review, external review, and multiple or 
mixed reviews. Most importantly, however, when the. review 
extends beyond internal program improvement, consideration 
should b^c^givcn to introducing greater degrees of objectivity by 
adding outside persons. In some cases, several different kinds of 
reviews may be put into operation with different- levels of 
responsibility for each. For example, in several states some in- 
stitutions handle parts of the review process and the state-level 
board takes care of other aspects. At whatever level the review 
takes place, certain critical questions should be asked. Will the 
reviews be conducted by the program faculty, an .interinstitu- 
tional . committee, state-level staff, outside • consultants, or 
perhaps a combination? What role will students play in the 
evaluation (see Smock and Bradenberg 1978)? And who will have 
primary responsibility: the college dean, a graduate dean, or the 
academic vice-president? Moreover, can this person play a mean- 
ingful role in planning and budgeting? And, ultimately, how can 
all the individuals and processes involved best work together? 
Who should serve on the review committees? How long should 
they serve, and what responsibilities will they have? 

Responsibility needs to be defined for each aspect of the review 
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system, with careful consideration given to existing organiza- 
tional relationships and review goals. In this regard, program- 
re\'ie\v cr ccria should relate directly to review purposes (Sparks 
1980). For example, if program reduction is a major goal, con- 
sider including specific criteria to address issues such as those 
noted by Davis (1981). While the task of identifying major 
criteria (cost, cjuality, need/demand, centrality to mission) for 
reviewing programs nationally may be fairly easy, finding ade- 
c|uate, accurate, and agreed-upon indicators for each criterion is 
more difficult.- Remember, entirely different indicators may be 
appropriate for different disciplines, and moreover, they should 
address adequacy as well as frequency and volume (see Lawrence 
and Green 1.980). 

Determination of how programs will be reviewed is, of course, 
intimately related to previous considerations. If peer consultants 
are to be brought in, keep in mind that they Ve unlikely to have 
an in-depth understanding of the local situation. Mechanisms 
need to be added that, will provide immediate perspective, such 
as the appointment of a local committee or other staff 
involvement. 

Each step in the proposed review process needs to be identified 
and the contents of each step determined. The resulting proce- 
dures wnW help determine the success of the review. By surveying 
various alternative approaches in use in other institutions, 
systems, and agencies, added perspectives may also be gained. 
Examples found earlier in this volume may further help in devel- 
oping appropriate procedures to fit local circumstances. 

Step Three— Developing Consensus 

To obtain consensus among those being reviewed, it*s essential 
to establish a viable mode of communication regarding the parti- 



.1 to "quality^" the work of Mnry Jo Clark (1974a; 1974b; 1977; 1979) is especially useful. 
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CLilnr review processes nnd procedures. Conceivably, this com- 
munication can range from simply disrrihurinc the proposal and 
asking informally for comments and suggestions, to setting up a 
committee for reviewing and making appropriate changes or 
recommendations. Whatever method is chosen, it should relate 
to the local political and cultural circumstances as well as to the 
degree of consensus sought. \n addition to involving those who 
are subject to the proposed review, it may be advisable to share 
the procedures with other groups, such as administrators, 
trustees, and legislators. 

If so desired, steps two and three may be combined into one 
step or even reversed in order. For example, an institution or 
state agency may want to appoint a committee (step three) to 
develop the conceptual approach to review (step two) .rather 
than present an already developed process to a committee or 
group of responders. Alternatively, the process and procedures 
^developed, in step two may be used as straw men for generating 
discussion leading toward a final product. Although these steps 
may be repeated several times, care should be taken to make sure 
participants don't feel railroaded into using a particular review 
system. 

Since it is unlikely that everyone involved will agree on all 
details, a point comes when those ultimately responsible for the 
reviews must conclude the consensus-making. If dissenting 
groups try to stall the process, they can sometimes be appeased 
by advising them that the entire process will be evaluated follow- 
ing the first round of interviews, or at a later stage. 

Step Four— Collecting Data 

Opinions vary about the best ways to collect, compile, and 
-verify data. Centralized data collection is preferred by some, 
since it can increase the likelihood of acquiring comparable data 
across all programs and may reduce staff reporting burdens. 
Others, however, contend that such units may actually end up 
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proliferating unnecessary data and, in the process, even incur 
qualitative losses. Whether data is collected by a centralized unit 
or by program faculty, careful verification is essential. Centralized 
data should be shared with individual departments, and con- 
versely, data collected by program faculty should be checked for 
discrepancies that may arise when tabulation is done by different 
persons with differing viewpoints. When data development is 
first he^nin, it should be emphasized that no data instrument is 
perfect: the goal is simply to develop evidence that meets the 
major rc .icw needs. 

Almost uniformly, survey respondents complained that, 
unless otherwise restricted, too much data was collected. As a 
result, reviewers tend, to end up with costly information 
overloads. According to Petrie (1978, p. 12), one solution to this 
is to **balance the amount of information collected with the kind 
and level of judgments to be made." And Poulton (1978, p. 6) 
adds to this by noting that **much of the information collected is- 
not necessarily new. The primary value lies in the assembly of 
diverse, information at one point, in one place, reviewed by par- 
ties in different locations in the institution, and subjected to a 
series of well-structured questions. . 

Some assistance in this regard can be gained by making use of 
various standardized survey instruments such as those provided^ 
by ACT and ETS. Many institutions and agencies u^ing 
standardized surveys either modify or add their own questions'to 
allow for open-ended responses,^ a process that is especially 
helpful in providing faculty with greater personal involvement. 
Such involvement need not detract from the usefulness of the 
standardized instruments and may even make their use and 
results more acceptable locally. Usually administrators insist on 
a common core of data from all departments for administrative 
use, which, for obvious reasons, frequently involves some give 
and take between both parties. 

One way to effectively classify and analyze data for decision- 
making is to use the NCHEMS Outcome Structure: An Overview 
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and Procedures for Applying It in Postsecondary Education Institutions 
(Lcnning 1977). This structure has been used by a number of 
institutions in developing outcome-oriented program-review 
data. Also, the Program Quality Assessment Measures project 
developed by the Council of Graduate Schools and ETS has pro- 
duced results that indicate that "individual student or faculty 
ratings can be averaged to obtain mean ratings that make useful 
distinctions between departments..." (Council of Graduate 
Schools 1979, p. 2). Although their application to other levels 
has "hot yet been determined, these results seemingly indicate 
that such ratings can be effectively used as a major component 
-.for graduate review. 

Of additional benefit to those conducting graduate-program 
reviews is the "Graduate Program Self-Assessment Service*' 
offered by the Graduate Record Exam Board and the Council of 
Graduate Scliools. Based on the results of the above-noted 
research concerning quality dimensions in doctoral education, 
this service uses confidential questionnaires to obtain judgments 
about doctoral programs from facylty, students, and alumni. 
Data summaries sent to participating departments or other 
designated units include tabulations of individual items and 
scale scores that represent judgments about departmental activi- 
ties in a dozen areas, A useful handbook for the development of 
program review has been prepared to complement this service 
(Clark 1Q80). 

Step Five—Selecting Consultants 

For a variety of reasons, external consultants must be carefully 
selected for the review process. No doubt there is more than a little 
truth to the common opinion that some consultants are willing 
to tell you whatever you want to hear. The following guidelines 
for consultant selection are based on the experience of our 
survey respondents: 
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• Use "outside'* persons for final consultant selection, but 
involve faculty in ways such as asking for names of possible 
consultants. 

• Try lo get a balanced, team representing the various 
schools of thought within the discipline or field. 

• Search widely for consultants Check references carefully, 
making sure consultants are qualified at the necessary pro- 

. gram level. . ^ . 

• Inform consultants at the outset regarding nature of the 
evaluation, salat tcs, and dates when reports are due. 

• Brief, consultants about the local environment and the 
specific outputs expected, without unduly influencing 
their ciecisions. 

• Insist that consultants spend an appropriate amount of 
time on campus and in the department under review. .. 

• Allow consultants sufficient time, support, and resources 
to carry out their tasks in a meaningful way ,"while centrally^ 
coordinating the visits. 

^Step Six — Conducting the Reviews 

At this time the actual reviewing takes place* Consultants 
(when required) make their site visits and the data is analyzed. 
Once verified, the data should be reviewed for preliminary con- 
clusions, and any additional data that-*may be needed should be 
requested. Because data interpretation can be difficult— exo^pt 
in cases of extremely good or extremely negative reports- 
statisticians or consulta'nts may prove helpful in explaining 
details. 

Step Seven— Assessing^the Data 

At this critical step the preliminary conclusions and recom- 
mendations are reached and shared in draft form with program 
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•faculty. Usually, however, this is done for. errors of fact and 
omission only. . 

The preliminary reyiew draft plus the exit interviews (if site visits 
are eondueted) provide program faculty and administrators with 
early i iieation of the major findings- and conclusions. By 
>: -vering errors and omissions, in time, the report *s final 
1. . bility may be preserved. 

Step Eight— Developing Recommendations 

Actions stemming from internal and external reviews can range 
from program expansions to program terminations. Because of 
this, recommendations stemming from the review-process need to 
be carefully drjifted, wMth due-process provisions included at all 
levels for those who may feel dissatisfied with the outcomes. If tlie 
recommendations have not been shared with those under review 
in the previous step, this should now be done. Ample opportunity 
should be provided for rejoinders and explanations whenever 
possible. External reviews, for example, may sometimes provide 
an opportunity for the institution under review^ to voluntarily ter- 
minate a program, hi such cases, opportunities for cooperative 
programs^ and consortia can be explored (see State University 
System of Florida 1979). 

During the appeal process, the credibility of the entire review is 
frequently given a close examination. But if the process has been 
well developed and fairly executed, there should be nothing to 
fear from this close examination. At all review levels, provisions 
for due process should naturally be made available. Within an 
institution, this can take the form of an appeal to.a higher admin- 
istrative level or to the board of trustees; at the state level, the 
appeal might go to either the commission, the board, or the execu- 
tive director. In some instances, the appeal process -at the state 
level may even include the unofficial step^^of appealing to the 
public or seeking legislative action. 
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Step Nine— Implementing and Using the Results 

According to Petrie (1980, p. 1), ''If program evaluation is to do 
anything, it must somehow interact vyith the other segments and 
systems . . . such as planning, budgeting, etc." Planning and bud' 
geting processes obviously help set program priorities; some pro- 
grams may be strengthened while others are reduced. In this 
way, recommendations stemming from reviews can help insti- 
gate institutional improvements. 

In this regard, Shirley and Volkwein (1978) have described the 
processes and criteria that enhance decisionmaking on academic 
program prioritiCvS, including external and internal inputs to 
decisions or priorities. And, in a similar vein. Lawless, Levi, and 
Wright (1977) suggest procedures for linking academic priorities 
to resource decisions, as do Micek (1980) and Munitz and Wright 
vl980). . [ 

Step Ten— Evaluating the Review Process 

< 

Unfortunately, the last step in developing program-review 
systems is the step most frequently ignored. Whether it's done by 
those conducting the reviews or outside consultants, the review 
process itself should be periodically reviewed to rnake sure that 
it's meeting the original objectives and adjusting to changing 
needs. Mims (1978, p. 2) remarks that "provisions must be made 
for amending the process based on unfolding needs and on ex- 
perience with implementation.'* Guidelines noted earlier in "this 
chapter may also he of some assistance. 

In a helpful handbook for evaluating aspects of program" 
reviews, Braskamp (1960,.p. 1 1) suggests that "utility" is 5 critical 
criterion for judging the worth and value of the review. Accord- 
ing to Braskamp, 'The information presented must also be 
understood, credible, and coherent to the intended audiences 
^and the evaluation must be a guide for politically feasible ac- 
tions.... If a two-way communication channel between the 
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cvaluators and lislm-s is established, provisions for trust and 
mutual problc: solving arc more likely/* 

Of the dozen or so evaluations reviewed in this study, all 
recommended continuance of the review process, with some im- 
provements or refinements. Obviously, developini? an effective 
review process is not easy and some will never be satisfied, but 
with periodic evaluations of the process itself, it*s possible at least 
to minimize objections while maximizing advantages. 
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Issues and 
Observations 



During the course of this study, I happened to visit a large 
public university where no less than half a dozen profes- 
sional accreditations were under way. System and state-level 
program reviews were in progress, a regional accreditation visit 
' had just been completed, and a self-study that had begun a year 
and a half before was in its final stages. Not surprisingly, the 
university president had recently informed the facc.;ty that he 
wanted to initiate a more systematic means for conducting in- 
ternal revievvs. Because of all of this activity, the provost was 
' worried thar camp'^s might become ovcrevaluated» In his 
words, there . > . ^ ded to be "some kind of rhyme or reason for all 
this program review activity.*' Somehow, '*tvvo or more of these 
evaluation efforts ought to be combined or eliminnced." The 
provost was not opposed to evaluations as such— in fact he 
viewed them quite favorably — but he wondered if the evalua- 
tions actually saved that much m5ney in terms of^the amount 
of faculty time and resources being spent on all of the duplicate 
activities. ^ ■ 
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Furthermore, he was openly concerned nbout the lack of par- 
ticipatikMi by private institutions in state-level program reviews. 
While his institution and the other public institutions were 
seemingly being reviewed constantly, the private institutions 
(many of which received state aid) wtM'e apparently completely 
free of review. **\V!iy/' he asked, '^shouldn't the privates also be 
accountable?** 

The provost *s comments are indicative of four key issues 
brought up by this study, namely: (1) the apparently unnecessary 
duplication of evaluation efforts; (2) state/institution friction 
about the ways program reviews are conducted; (3) the involve- 
ment of private/it'idependcnt institutions in statewide reviews; 
and (4) whether or not program reviews do, in fact, save 
money. 

Duplication of Evaluation Efforts 

Understandably, the is-uv of duplicative evaluation is heard 
discussed most frequently at the institutional level, where it hits 
hardest. Although duplicate evaluations in and of themselves 
may not be a problem, when they complement each other to 
the extent that they are unnecessarily duplicative, complaints, 
of counterproductivity bear thorough consideration. Several 
solutions to the issue of unnecessarily duplicative reviews were 
suggested at the institution/state, institutional/ accreditation, 
and state/accreditation levels. 

Most often, respor\dcnts suggested combinii». . '^^everal evalua- 
tion efforts into one. This solution sounds t^oc.is ; j)^h, assum- 
ing that all of the various program-evaluauo;. eitor ate 
mutually compatible. However, both theorc'ja' : i^l pia- ticai 
evidence says that they usually are not (svo 'ig;zr- J 6). On a 
theoretical level, an important distinction cwjrges b.^t'vveen' 
formarive and summative evaluations. Where institutional pru- 
"g.am reviews are usually formative in the sense that they seek 
program improvement, system-level (and especially state-level 
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reviews) arc generally suiiinuuivc, since lliey Iiave the piHentiJil 
i>f leading lo program termination. On a practical level of im- 
pleineiuation, other distinctic^ns arise. Because institutional 
re\'iews are almost always aimed in voluntary self-improvement, 
they tend to skip over the very ciuestioiis that external 
e\'aluators are most keenly interested in. 

To give an example, at the institutional level "need" is fre- 
quently defined in terms of student and/or faculty needs, while 
at the state level, *'need" is more likely to be defined as either a 
societal or manpower need. Similarly, ''efficiency" at the institu- 
tional level has very different connotations than at the state 
level, where it most likely will concern cost comparisons between 
institutions, as well as unnecessary duplication. To resolve these 
differences, some advise setting up a system for independent 
institutional evaluation by "trained neutrals" (Harcleroad and 
Dickey 1975, p. 3) and Harcleroad {1^^7()). Aliernatively, some 
suggest expanding the triad concept of institutional licensure, 
evaluation, and accreditation to include mechanisms for pro- 
gram re\'iew (Kaplin 1976). 

Despite : recent trend by some state agencies to include or at 
least recognize institutional review efforts, a strong feeling per- 
sists at many institutions that such accommodations are 
counterproductive. In their views, program accreditation in itself 
should sufficiently satisfy state-level inquiry.' A response to this 
contention, however, is found in Griffiths (1979), who notes: 

In accreditation, the main question is: What is the program 
trying to do and how well is it doing it? [But] in program 
review, the question is: Does the program meet or show clear 
potential for meeting standards of high quality and demon- 
strated need regar dless of what the program wants to do? 

Even when such considerations are taken into account and 
program-review efforts are effectively combined on campus with 
regional accreditation visits, at the state level the problem is fur- 
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tluT complied t\l diK' to a frcquciuly acknowledged animosity 
IxMw'orn stall' ai^cMuics and both nationJil and accreditation 
groups, Piirtly because ol this, some site \'isitors lor accreditation 
purposes ha\'c lound it difficult, if lu^t impossible, to serve two 
masters when suite and accrediting efforts are combined. 

In more tbiin one state, advice was given by state-level staff to 
'*keep accreditation agencies out of tbe review." In fact, a few 
states eitber have threatened or are threatening to refuse 
recognition of regional jiccreditation efforts altogether. At issue 
here is a beadH)n conflivt between the state and pri^fessional 
accreditation standards, such as those that prescribe how an 
institution must be organized. 

An attempt at reconciling these differences is posed by Tucker 
and Maut: (1978), who, writingin tbe Educational Record, suggest 
involving state board staff in regional accreditation visits. Pres- 
ent evidence, in fact, indicates that ^uch an excliange is begin- 
ning to take place. For example, more than 80 percent of tbe 
c\aluarion teams surveyed in the Middle States Association 
region said they now enlistHhe support of state board represen- 
tatives (Kelly 1979). Awl in some states, such as Texas and 
K4arvland, the state coordinating board and tbe regional accred- 
iting organization are actually combining institutional evalua- 
tion efforts. As a part of its policy called "Higher Education 
Evaluation and Development'' (HEED), the Maryland State 
Board for Higher Education agreed to make cooperative team 
visits with certain colleges and schools to reduce the "heavy and 
duplicative burden of self-study and planning activities" (State 
Board for Higher Education 1979, p. 3). 

Another familiar suggestion for reducing unnecessary duplica- 
tion of sote and institutional efforts is simply to drop one or tbe 
other evaluation. But for various reasons, this is rarely satisfac- 
tory. While it may be true that not every institution, system, and 
state-level agency should be continuously reviewing all programs, 
it is bard to figure out where to cut back on these efforts. On one 
level, most survey respondents indicated that institutional 
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review.^ were not only ncccssnry but ought to he comprehensive, 
re^uliirly scheduled, ;uk1 ii pnrr ofn comprehensive institutional 
phuining and budgeting system. On another level, stnte-level 
reviews are increasingly taking on added significance in light of 
preseni political and economic climates. In many areas of the 
cmnitry. external reviews are considered essential to reassure 
state olficials and the public that higher education is accountable 
(Geiogue 1^)80). Governance patterns, the number of institu- 
tions, iuul the size of the state all enter into the decision of when 
tv) undertake reviews at the extrainstitutional level. All things 
considered, the results of this study iiulicate a need for both in- 
ternal and extrainstitutional reviews in virtually all instances. 

State and histitutional Friction over Reviews 

Aside from issues surrounding duplicate review efforts, prob- 
alily the bigjiesr obaav h.' to cooperatioi^. is an attitudinal one. 
College and university faculty and staff openly resent the 
increasing state intrusions made into academic autonomy. To 
case such tensions, the Carnegie Council, the Education Coai- 
mission of the States, the Sloan Commission, and others have 
attempted to make distinctions between legitimate state 
responsibilities and strictly institutional affairs. The Sloan Com- 
mission, for example, recommends that each state arrange for 
periodic reviews of every program but adds that these reviews 
should be conducted by peers, not state-agency staff (Sloan 
1980).- 

DespiU' such efforts, little consensus exists as to where respon- 
sibility for program review should ultimately reside. Meanwhile, 
concern over recent academic scandals, the seemingly endless" 
requests for more funds, and the inappropriareness of at least 
some university research press state officials to ask for ever- 
tighter controls and greater accountability. Perhaps understand- 
ably, program reviews tend to be increasingly popular with 
governors, legislators, and the public. 
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Perhaps I he following cdiiorial concerning reviews conducted 
by the Louisiana Board of Regents is typical: 

Cnrefullv nurturing the flourishing programs and cutting out 
the obvious losers, as painful as the latter might be,. . .is the 
way to go for a better, higher education. Just be grateful some- 
thing is being done here to assure that vveVe getting a full 
measure for what we pay. It's certainly a refreshing cliange 
(Morning Advocate 197^)). 

When emotions come into play, program review is viewed as an 
impc^'tanr tool (for both right and wrong reasons) to make insti- 
tutions more ^iccountable, to weed out duplicate programs, to 
reallocate resources, and, in a word, to cut costs. 

Participiition of the hidependent Sector in Program Review 

Despite the fact that most state agencies feel participation by 
private institutions is an absolute must, only a few agencies liave 
authority that actually extends to the private sector. Especially 
in cases where state aid is provided and reviews are integrated 
into state planning and budgeti ng systems, lack of input from the 
private sector becomes critical. Obviously, without a reciprocal, 
opportunity for state agencies and public colleges to scrutinize 
private-college offerings, it's virtually impossible to provide truly 
comprehensive statewide planning perspectives. Although on a 
voluntary basis some private colleges and universities do occa- 
sionally participate in state-level reviews, to date, this participa- 
tion is limited to only a few states. 

By and large, private institutions fear that participation in 
reviews will infringe on their academic independence. But while 
these institutions may understandably sacrifice some of their tra- 
ditional autonomy, one WTiter has commented that "the benefits 
outweigh the costs of intense programmatic competition and ter- 
rific resource waste'' (Wilson^ 1979, p. 20). This conclusion not 
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o\\\\ h.'is hccu substJintiiitcd by the findings of our iiiimalintc 
study Imu iilso is i"i>usistcnt with conclusions rcmhcd hy the 
Skinn (^nniuissit)n (l^WO). For i\ C(Miiplctc nnnlysi^ of privntc- 
sector pjirlicipation in siatc-lcvcl reviews, sec Wilson atid Miller 
(l^)SO). 

Ho Progrjun Reviews S;ive Mtmcy? 

C^f all the concerns raised in this study, no issue received more 
tretjucni conunentary than the issue of whether or not program 
^lisconti nuance (especially as a result of state-level reviews) 
results in dollar savings. When the author and Rohert Derdahl 
lirst wrDteon state-level reviews, a common folk wisdom prevailed 
that program reviews would naturally result in great doHar sav- 
ings. Later, this was replaced hy aii equally common hit of lore 
that you canY save money hy eliminating programs (see Berdahl 
1975). As in many issues, the truth probably lies somewhere in 
between. Documented cases for bcuh positions can easily be 
found (Smith 1975; Fields 107(^). Mingle (1978, |). 66), for exam- 
ple, in a study of state-level program review in the South, found 
that program re\*iew is probably not a "cost-cutting measure if 
[emphasis added] we are considering imiwdiatii cost savings," 
According to Mingle, "States which use quantitative measures of 
degree output t^ften limit their reviews; to programs which are 
already low cost. |But| the most likely place to find savings is in 
programs with high levels of enrollment and degree production, 
wliich ciuantitative evaluations exclude/' 

Wl- f her or not savings ultimately accrue depends on a variety 
of facu^rs. For example, what is the primary purpose of the 
review? Is it to save money or is it for some other reason? And 
lunv extensive is the program beii; eviewed? In other words, is 
it simply a paper program or is it a genuine program with large 
numbers of faculty, students, and equipment? 

To dare, much ofihe evidence is not yet in. This is particularly 
so where resources are reallocated to other programs and may, in 
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clTci.t, rtMiiiiiii hiLlJcn. Furtlicnnorc, in sotnc iiistitutioiis nnJ in 
most stiitt's, tin* first nninJ of proi^rnm clitniiintioiis has foi:uscd 
on I lie \vi'akt.'si programs. I liosc programs u'ith declining 
cMirollnKMits, poor iiuality, and lack of institutional support were 
autotnatieally piekeil first. Sonic, ii^ fact, were juiper programs 
without either students or full-time (acuity. As the reviews ex- 
pand, as enrollments continue to decline, and as new resources 
Ivcome increasingly scarce, reviews will necessarily select more 
suhstanti\'e programs (ov elosurc, thus making dollar savings 
more e\'ident. Nonetlu'less, reviews themselves can he costly. If 
done well, they consume Inrge amounts of lime, energy, and 
money. But ilespite the expense, survey respondents concluded 
in virtually all cases that the r*eviews are, indeed, worth doing. 

At this point, it's virtually impossible to forecast the future 
agenda for program review. The facts to date seem to indicate 
that— with the possible exception of the cost-saving issue—the 
inherent conflicts will most likely persist. At the institutional 
level, the confliet will no doubt continue between faculty who 
would like to utilize review outcomes for program improvement 
anil aclministrators who wish to use these same results for 
resource allocation. And similarly, the inherent conflict between 
institutions and external reviewers at the system nnd state levels 
will probably continue to be a further source of friction. Some 
states have been more successful in maintaining this friction at a 
healthy level than others, and hopefully their examples miiy 
serve as guideposts. Obviously, we are now well past the point of 
turning back the clock to the relatively itum c relaxed eras of the 
' fties and sixties. State pinnning is very much a part of higher 
education with little indication of a'natement. And accordingly, 
academic program evaluatio!v~b'^' h internal and external—will 
he much used and discussed tr roughout the decade of the 
eighties. 
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Terms and 
Definitions 



A brief review of evaluation terminology as dcYived from 
the literature can help inform the reader about current 
trends and applications. To start, evaluation itself has been called 
*'an elastic word that stretches to cover judgments of many 
kinds'' (Weiss 1972, p. 1). As such, it can be used to describe a 
host of activities pertaining to the cpmmdn notion of judging 
merit. For the purposes of this report, two particular kinds of 
evaluation are' brought into direct focus: program approval, 
which offers a process for evaluating new program proposals, 
and program review, which provides for evaluation of programs 
chat are already in existence. (The reader is referred to Kelly 
and Johnston [1980] for a more thorough discussion of program 
review and its relationship to evaluation.) By way of contrast, 
the type of evaluation known as accreditation can be seen a? "the 
process by ^ /hich a program or institution is recognized as being 
in conformity with some agreed-upon standard. Most frequently 
it refers to approval by voluntary associations and accrediting 
agencies" (Anderson and others 1975, pp. 4-5). 
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A lUiM'c precise understanding of the terms "program 
appri)\'ar' and "re\'ie\\'** can he gained hy Kx^king closely at the 
context in which they are often used. Fov example, wl^en these 
terms are employed hy system or state-leveli agencies, a notion 
i^f acciumtahility generally accompanies them. Acconntahility 
has heen \'ariiH,isly defined as "respi^nsihility, explicahility , and' 
answerahility** when used with reference to service in the public 
interest; in the field of education, accountability ^'relates mainly 
to a concern for furthering the educational effectiveness of 
school systems** (Anderson and others 1975, p. 1). 

Two related ciMicepts are ci.s.ses.smcnt and critical jucl^^nient. 
A.sM'.s.snicnt connotes "a narrower meaning than evaluation, but 
a broader meaning than measurement/* It refers '*to the process 
of gathering the data and fashioning (it) into an interpretable 
form , . (Anderson and others 1975). Critical judgment, on 
the (uher hand, comes at the. final stages of the assessment com- 
ponent, when decisions are made regarding the future of the 
program being reviewed. These decisions can range from either 
termination or continuance, to continuance with conditions, 
„La_.pto.gcaui-„ex{mnsio.n_andiurdn c r dc ve 1 a\m-\ cii : ^ 

Pro^Tcnn in this report generally refers to an academic/ 
occupational course of study leading to a degree or certificate. 
Certainly program-review activities, however, extend beyond 
~~just dcgrcc and certificate programs to include research and ser- 
vice programs as well as projects such as the College Outcome 
Measures Project (COMP)and the Academic Program Evaluation 
Project (APEP), which evaluate entire undergraduate programs. 

Objectives of program review and approval generally focus on 
program improvement and efficiency. On the institutional level, 
emphasis is primarily on improvement — with efficiency and 
effectiveness today becoming relatively more important 
—whereas at the system and state level', emphasis is more likely to 
be placed on the efficient use of state resources. Admittedly, 
qualitative improvement (usually in a broader statewide sense) is 
also an objective in many state-level reviews. Whatever the case. 
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these i)hjLVti\'L\s iirc mot Lluriii^ the rc\'ic\v :uk1 a[ipr(U';il pruLcss 
hy the use i^f LritL'Mo?! iiwasurcmcnt, whieh seeks t(^ rehue 
;uk1 ohjee(i\es til ;i set iifestuhlisheJ eriteria (AudersiMi 1^)75, pp. 
^)8-100). 

A further distiiu tiDU is th;u iviiKle between foriiKUive [\nd sum- 
miuive e\':iki;uions (Scriveii 1*^)67). Basieally, j^mmhv evakia- 
tion prtKluees iiifi)rin:uii)n that ean he used form a program lu* 
help improve it, whereas ^ummcuivc evakiation sums up or 
evahiares the overall piogram once it is in operation. Summative 
evaluatinn, ft)r example, may result in program modification or 
terminatiiMu Generally speaking, reviews dune internally hy 
program faculty are formative, whereas reviews donehy outside 
individual:-^ are sumiiuUive. More specifically, internal reviews 
conducted exclusively for the use of a given department are 
located near the formative end of the continuum; reviews con- 
ducted at the institutional level arc slightly less formative; profes- 
sional accreditations arc somewhere in the middle; and external 
reviews conducted by system-level or state agencies are near the 
summative end (Barak 1^)77). 
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Examples of 
State-Level 
Program Approval 



The following examples indicate the various approaches to 
state-level program approval. Virginia represents the 
-pl-i-npviFvg^i-ype-approaGh r R-hede-Isla nd-exei^^^ 
mental approach, and Washington combines both. Beyond 
illustrating rhe various approaches to the state-level approval 
process, these three examples indicate the significance presently 
attached to srate-level approval in initiating new programs.- 

Planning-Type Approach: Virginia 

In 1978, the Virginia Counci 1 of Higher Education (SCHEV) 
began asking for (and reviewing) all public institutional curricu- 
lar plans. In the hopes of developing a continuing and compre- 
hensive plan for the introduction of new academic programs, 
SCHEV asked that these plans be revised every two years. 

Since the council recognized at the outset that starting new 
programs required cither additional funds from the general 
assembly or else reallocation of funds from programs already in 
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opLM'atioii, all iH'W McaLlcinic proposals were ro he coiisii^lcrcd in 
lii;hr o( projoeiL\l enrollnuMU fijiurcs, operational huelgets, and 
hiiilLlinjL^ needs of colleges and universiiies (Davi<.'s 1^)80). 
Furrherniore, to get ai^proval, ciu li institution^ comprehensive 
six-year curricular plan would noecssarilv incorporate approval 
by SCHEV o( all specific academic programs contaitied within 
it. Onl >' under utuisual circumstances could an institution 
either request permission to alter its curriculum plan or appeal a 
council decision. Above all, the SCHEV plan was designed to 
curtail imnecessary budget expense, avoid program duplication, 
and simultaneously find alternative ways to nuike essential 
programs available to the public. 

hicremental Approach: Rhode Island 

The Rhode Island Board of Governors supervises all state 
public L^ducation. Altliaugh proposals may be submitted at any 
time, the staff of the commissioner of education asks for ade- 
quate time for study and analysis. Because of this, proposals 
must be submitted at least 90 days prior to the intended date of 
implementation. 

A primary consideration of Rhode Island's review process is 
to find out if program duplication is involved and, if so, 
whether the duplication increases costs. In other words, to 
what extent can the,prop9sed acdvity utilize available res6urces 
at other institutions? Toj adequately respond to such queries, 
program proposals must include answers to the following; What 
cooperative arrangement's will. be made with institutions offer- 
ing similar programs? Will provisions be made for students in 
other Rhode Island institutions to transfer into or out of the 
program? And if the program happens to be in an occupational 
field, what are the curfert and projected manpower needs at the 
national, state, regional, and local levels? In a woid, how large is 
tbe projected clientele, and to what extent might it conceivably 
draw students from other existing programs? Specifically, how 
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will the jTrt)pc)SL\l |M-t)^rani be ;alininisUM'i\l iiiul hcnv mueh 
will ii <:os{ I 

eninpleiion of the review, ihe vDinmissicuier suhiiiils w 
I'ev'oiiiiueiRlntioii lor aeiion lo w S|ieci;il sulH'(Miimittee for 
l^ostseeoiKhiry eJuviUion, whieh in luni |)reseius its reecunmen- 
viations to the hoard k)f ^^overiiors. Ajij)roval by the h(Ku\l (^f 
Ho\eriu)i'S is |Tralie;Ued on the nssuminion thiu \m institutional 
evaluation of the new j^ro^ram will take jilaee ow m* beftu'e the 
completion ol one full j^ro^ram eycle. ' ^' 

(\imhinevl AjijM'oaeh: WashiiA^ton 

The Washii i^ton Couneil combines the ineremehial approach 
with the i^lanning npproach in such a way as to coordinate j^'o 
gram apj^rcn-al with the legislative budget cycle. The following 
flowchart (figure 17) illustrates the normal setjuence of events 
from conception to implementation of a new program. After 
deciding tt^ plan a new degree program, the institution enters 
into informal discussions with council staff to determine whether 
the pr(^p(vsed |')rogram will reijuire a jireliminary (or category 1) 
planning statement. This i^reliminary planningstatement is then 
reviewed by staff and forwarded to the institution. 

To ctuncide with the budget calendar, the main stage of the 
review must begin on or before July 1 for inclusion in the normal 
review agenda. Final action by the council must occur on or 
before December 15. If the council makes favorable recommendii- 
tions, stipulations arc included that the program must undergo 
institutional evaluations no later than the fifth year of its opera- 
tion. At that time, new evaluations will be compared .with the 
original projectituxs to examine the overall impact of the pro- 
gram. A final evaluation report is then sent to the cc:)uncil, con- 
taining the findings as well as any necessary measures to improve 
the program. 
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Washington Council on Postsecondary Education 
Procedure for Review and Recommendation of 
New Decrees or New Decree Programs 
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APPENDIX' 111! 



Internal Review 
Examples 



Although no particular approach to internal program review 
is recommended as perfect, the following examples provide 
an overview of current procedures, The University of Iowa ex- 
ample indicates how a large research university approaches the 
subject, the three community colleges illustrate the variety of 
processes found in smaller institutions, and Harvard University 
represents a highly uniciue approach based on the school's 
history and traditions. 



University of Iowa 



Although typical of the approaches found in many large 
research universities, the University of Iowa*s internal program^ 
review process is especially noteworthy because of its integration 
into long-range academic planning. Initiated in 1971, the UI aca- 
demic review process enables the university to systematically 
review all programs and levels, including individual support 
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units as wvW as iIk* vinivorsity as a \vhoK\ hnr I'xainpli', rcsiihs ol 
dL'pariiucmal aiul |j;radunii' prn|j;ra!n ri'virws arc iiKorpnrau^l 
into ihr t)\x'rall rollc^iau* rev iew of prinriiies aiul Jireciiniis, 
iisin^ llu; t;uidi'liiu's lor 'Volk'^iiiu* reviews." These collegiate 
iliiijeline'i then serve lor review of such uoiule[>arinieiuali:eil 
eolK'i^es as law. luirsing, aiul phariuaey. AikI linally^ in eonjune- 
lioai Willi the Klorth C'entral Association acercilitat ion visits, a 
.total university review is conipleieLl. 

Above all, this plan is JesigneJ to allow flexibility in iiieetinij; 
indi\ i\.lu.,li:eil pro^rnin needs and characteristics. To ^ain a bet- 
rer understanJing of the specific applications, we'll turn now to 
the separate processes involved in departmental reviews, col- 
Ic'f^iate reviews, and the university review. 

Departmental Program Reviews 

Basically, UTs departmental reviews aim at gaining quality 
control while surveying the overall future direction of the aca- 
demic enterprise. At t:he heart of UTs departmental review is a 
self-study that focuses on goal statements, restiurce requirements 
for attaining these goals, and comments by the deans t)n stated 
needs. 

For each departmental review, an ad hoc review ct^mmittee is 
appointed to interview students and faculty, enlist reactions 
from external scuirces, and evaluate teaching, research, and ser- 
vice functions. Resource persons from outside the university 
serve with faculty from other university disciplines on the review 
committees. Among other considerations, the review committee 
members try to gain an understanding of the relationship of the 
program or. department to other college programs and units; to 
evaluate faculty contributions;, to review program goals in the 
context of projected needs, as well as existing financial and 
mnterialsupport; and to identify strengths and weaknesses in the 
program. 
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('opii's ol tho liniil rov'icw report nrc suhiniitcLl to the apprv^- 
priiUc Lk'iuis, \\'lu>, ill turn, suhniii their reconuiK'nthu ions to the 
^•iee-presiJeiu lor juiKleniie nffnirs. 

( Aille^iate Re views 

(.]olle^i;ite review eonunittees nre nppointeLl hy the viee* 
presiilent for nciKlenue afliurs JinJ eonsist of fneully from other 
lUiiversiiy eollejijes plus at least one person from outside the uni- 
versity. The aetual review makes use of a self-study prepared l^^y 
the eollef^iaie faeuliy hut foeuses primarily on the mission an^i 
pritM'ities ol the college as a whole with respect to teaching, 
re..earch, anil service programs. In addition, the review evaluates 
the collegiate organization, administration, and fiscal structure. 

After the review committee obtains faculty self-studies, reports 
from the dean, plus additional information, it then prepares a 
formal report that is submitted to the vice-president for academic 
affairs, who then consults with the dean, the president, and an 
appropriate college faculty conimittee. 

University Review 

Begun in the fall of 1976, the university ^eview surveys five 
general areas: 

1. Undergraduate Education. Issues are general education; 
student advisory services; job-oriented programs; inter- 
disciplinary teaching programs; honors programs; anci use 
of teaching assistants. 

2. Prcprofessional andProfessional Education, issues are inter- 
' relationships of teaching, research, and professional prac- 
tice; professional education and the University; prcpro- 
fessional advisory services; and professional accreditation. 

3. Graduate Education. Issues arc faculty vitality; employ- 
ment of graduates; new graduate programs; inter- 
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Llisciplinary research; and libraries, computer resources, 
aiul other research vSupport services. 

4, Contiruiiii^ ELlucation. Issues are professional growth 
programs; inservice proji^ranis for teiichers; relations with 
areii'coniniunity ct)lleges; off-campus Jej^ree projj;rams; pro- 
grams for adults; and recreational and cultural pro^',rams. 

5, Governance^ Faculty Matters, vStudent Affairs. Issues are 
faculty organizations; student organizations; teaching 
evaluation; academic reviews; affirmative action; and the 
role of students in acadea^ic affairs. 

livalua.tion of UTs Acaden^c Review Piocedures 

huerviews with executive officers of departmeius and pro- 
grams at the University of Iowa that were reviewed between 1971 
and 1978 revealed that faculty who served on review committees 
from other departmental units felt that they had gained a 
broader understanding of the university. Tl'iey alsco felt that, 
despite the significant investment of faculty and administrative 
time and effort, many .positive benefits had been derived for 
academic program evaluation and development, especially as a 
result of tiie self-study prc^cess. The interviews further indicated 
that procedural guidelines for academic reviews needed greater 
clarification and refinement, and that improved follow-up 
mechanisms were needed to enhance the overall review process'. 



The Community Colleges 



California Community Colleges 

COPES is the familiar acronym for the Community College 
Occupational Programs Evaluation System, a process that was 
developed by the chancellor's office of California Community 
Colleges (Sacramento) and Foothill— -De Anza Community 



104 



1 no 



inii;knal iu;vikw examples ^ 

College l^istriii (California) ti) evaluate cK*ciipatinnal education, 
Between 1^)76 and 1977, 53 evaluatii)ns usin^ COPliS were 
undertaken in C^nlifornia Conuuunity C^olle^es. Aect)rding to 
the findings, altluni^h a nuniber of peo|)le saw need for further 
refineiuenl, the overall reaction was favorable. Anions the 
acknowledged benefits rept^rted by institutions usin^ COPIivS 
were: better larj^etinj^; i)f fund areas for needed eciuipnient; 
clearer rtiles in administration and* program operations; and 
stronger plaiuiing, evaluation, and student follow-up. 

The way CX')PIiS works is by offering a self-study method that 
can readily he applied to a variety of siruarions, First of all, 
CX')PliS can be u,sed tt) appraise a college's total (XM-formance in 
iH'Cupational education or it can concentrate on a single 
program area. Also, it can evaluate specialized programs and ser- 
vices such as those offered for the disadvantaged and the handi- 
capped or cooperative work experience projects. All infill, the 
COPliS approach involves teachers, counselors, administrators, 
students, and occupaticMial education advisory committee 
members. And in some instances, a team of outside experts from 
cuher community colleges visits the camj^^us to validate the find- 
ings of the self-study. Primarily, the COPj[iS process offers a ttx^l 
for use by the requesting college, the results t)f which are not 
reported beyond the campus. 

Broome Community College (New York) 

A second model of program evaluation for two-year colleges is 
•the RBE (Reality-Based Evaluation for Two-Year Occupational 
Programs) developed by the Cornell histitute for Research and 
Development in Occupational Education. Since Broome Com- 
munity College (in Binghamton, New York) uses an adaptation 
of the RBE model, it serves as an appropriate example. A key 
asset of this approach is the flexibility it allows in acquiring infor- 
mation for improved decisionmaking in areas such as standards, 
goals, and 'expansion or curtailment, while simultaneously de- 
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iMnphiisi:in^ iiKli\'iJiial pcrlornuiiuw \Vi(h ihv RBI: system, pro- 
^riiin ov;ilii;uii)n is Jimu* by ilu* Ji'pjirnnoiii i hiiirpiM'sori progrnin 
aHM'cliiKitiir iind by faculty whn arc assisted hy the clean ofcur' 
riculmn. As surh» it prtuceds ihroui^h three general phasesi^, 

Phase I seeks ID etarify prt^^rain objectives and activities. Tculo 
this, ail evaluative \vt?rksheet is Jevclopetl to look at such aspects 
as enrollment patterns, instructional technit|ues, course ohjcc* 
tives, anci prt)jectecl pri^gram needs. Ifconcerns arise during the 
evaluation (suc-h as a need for better evaluative strategies or a 
problem with lueeting proj^ram expectations), a committee is set 
up to do an in-depth study. Otherwise, Phase II bej^fns. 

Phase II is the data^eollection part of the process, which entails 
^atherinjH informaticMi about eacIV objective and/or cluster of 
activities as well as evidence related to the attainment of the pro- 
i^ram objectives. At this point, the Jean of curriculum and the 
department coordinator put together a .^'cond evaluative work- 
slieet to determine the most appropriate evaluation sources, 
such as faculty, graduates, students, emi^loyers, and st) ou. To 
find out which sources are, in fact, most suitable, the following 
questions are asked: What evidence can best determine attain- 
ment of the objectives? Where can it be found? How can it be" 
acciuired? 

Phase /// focuses on the findings and recommendations when 
the data is collected. Actual interpretation t)f the findings and 
recommendations is made by the dean of curriculum after the 
department chairperson program coordinator has forwarded a 
summary to the dean. Mainly, this report is concerned with pnv 
gram effectiveness, necessary modifications, findings of merit, 
and unanswered questions. 

Finally, a recommendation report is written up in the following 
format: 



106 



INTERNAL REVIEW EXAMPLES 



College Name 
Evaluation Report Title 

Identification— — ' College' 'Mission-Starement; Depart-"" 

ment or Program Goal; Program Title 
Introduction— Brief summary including -purpose of 

~~ evaluation and names of those involved 

in evaluation. 

Overview— Brief description of the process used, 

including objectives and activities of 
. the program. (A copy of the Phase 11 
worksheet may be sufficient. 

Findings— A summation of the findings, answer- 

ing the question. Now that we have 
evaluated, what have we learned? 

Recommendations— Answers the question. What do we do 

with what we learned? 

Metropolitan Community College District 
(Kansas City, Missouri) 

The Metropolitan Community College (MCC) District has 
been selected as the third community-college example to illus- 
trate a vocational curricular evaluation model. This model was 
originally funded by the Bureau of Occupational and Adult Edu-' 
cation of the U.S. Office of Education. The project primarily 
sought to evaluate the MCC program in six areas: (1) program 
relationship to the job market; (2) level of community support; 
(3) program success in meeting vocational aspirations of clien- 
tele; (4) program success in terms of student performance; (5) pro- 
gram cost-effectiveness; and (6) program success in reaching the 
handicapped and disadvantaged. Ac cording to its dev elopers, an 
lidvaHtap'oTltlTfsTnoHen^ of 
procedures for assessing the quality of existing vocational educa- 
tion programs at any postsecondary institution. 

For the actu al project, each of the above criteria vyas opera- 
tionalized in terms of subcriteria evolved from MCC evaluator 
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responses to a series of questions. A weighting scheme was then 
Jlicorporated iiuo* the model, reflecting the perceived impor- 
tance placed on each of the criteria and subcriteria by an ad- 
visory committee consisting of 121 state and local legislators, 
educ-citors, college trustees," and business people. The resulting 
proportional weights of the subcriteria are shown in figure 18. 

Once the evaluation criteria and subcriteria plus their respec- . 
tivc weights were established, a means for measuring a program *s 
level of success in attaining the subcriteria was developed. To do 
this, proficiency levels for each subscription were established 
that categorized a program's subcriterion attainment as strong, 
adequate, or weak. 

^ The criterion and subcriterion weights were then combined 
with attainment ratings on an evaluation score sheet (see figure 
19). A final step in the process was the development of evaluation 
profiles. 

FIGURE 18 

Paired-Comparison and Proportional Weights 
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FIGURE 19 

Criterion and Composite Scores of the Ten Field Tested Programs 
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Harvard 



Harvard University offers one of the oldest and, perhaps, most 
traditional approaches to program review. Par*:ly due to its 
historical evolvement, the Harvard approach is probably the 
most difficult for other institutions to adopt. At the heart of Har- 
vard*s review system is tho Harvard Board of Overseers, which is 
one of the two central governing boards of the university; the 
other is the president and fellows, commonly known a.s the *^cor' 
poration." The board is comprised of 30 members — all alumni by 
custom— whose primary duty is "to keep the University true to 
its character as a place of learning." Although in the very broad- 
est sense the executive management of the university rests in the 
president and fellows, it is the job. of the board of overseers to 
review and sanction the acts of that body. 

With the growth of the university in size and complexity, the 
overseers have had to seek help both in evaluating effectiveness 
and in providing support and counsel for the president and 
fellows. In this regard, they rely heavily on visiting committees, 
which are primarily *headed by overseers. These committees are 
made up of experts from outside the Harvard faculty or adminis- 
tration who are asked to evaluate a particular department in 
sucii a way as to bring fresh ideas to the university, to prevent 
provincialism, and to "bring the University ynto direct and 
active sympathy and communication with the outside 
world . . . By way of explanation., "department" in this sense is 
broadly interpreted to mean a school, museum, institution,.or, 
of course, a department. One further purpose of the visiting com- 
mittees is stated in an 1889 committee report that notes that 
every "really useful" visiting committee must "always remain a 
continual applicant for means." Members of the visiting commit- 
tees are therefore encouraged to help raise funds, or if they are 
able, to contribute directly to the activities of the departments 
with which they are affiliated. 
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INTERNAL REVIEW EXAMPLES 



Once a review is undertaken^ visiting committee chairmen are 
expected to report orally to one of the following standing com- 
mittees of the board of overseers: natural and applied sciences, 
social studies, humanities. and arts, or student life and financial 
policy. For example, the Committee to Visit the Department of : 
Physics reports to the Standing Committee on Natural and 
Applied Sciences, and the Committee to Visit the Department of 
Government reports to the Standing Committee on Social 
Studies, and so on. Tlie several standing committees^, in turn, 
report to the full hoard, which then brings matters of particular 
significance to the attention of the president and the 
corporation.^ 



'F(«r tSosc who wish further L'\;nnplos o\ pro^nini i.*v;iUi;uiiit^ in suuc idIIcj^cs :ukI universities, in- 
tormr.tion iim he loituJ in ;i |MjlMiv;itii)n of the ReSDUree Center (or PliitineJ Ohnn^e of' the 
Amerii nn Assm intinji o\ Stnte Colleges ;inJ Universities (AASCU) entitled Pra^rum hiiiluiuian 
(N'ovemher h^T^)- Also, n more recent pul^heiition by Munitz ;ind Wright (H\SO) cleserihes the 
evahintion pn)eessuseJ :u Miehii;;jn Stnte Uiiiversity, the University uf Miehi^nn, JinJ the Uni- 



■\'ersitv of HoustoiTTiil 


:o see Crnvcn mO). - 
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